Oct 30 21:58:22.000460 ip-10-0-141-194 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:58:22.000469 ip-10-0-141-194 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:58:22.000475 ip-10-0-141-194 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:58:22.000724 ip-10-0-141-194 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:58:32.020803 ip-10-0-141-194 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:58:32.020826 ip-10-0-141-194 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fd8771b688154c058996e69a1525746d -- Oct 30 22:00:36.291488 ip-10-0-141-194 systemd[1]: Starting Kubernetes Kubelet... Oct 30 22:00:36.711490 ip-10-0-141-194 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:00:36.711490 ip-10-0-141-194 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 22:00:36.711490 ip-10-0-141-194 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:00:36.711490 ip-10-0-141-194 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:00:36.716219 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716112 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:36.716219 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716198 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:36.716219 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716218 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:36.716219 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716223 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716228 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716233 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716237 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716242 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716246 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716249 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716253 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716257 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716261 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716265 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716269 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716274 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716278 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716282 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716286 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716291 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716295 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716299 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:36.716477 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716306 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716313 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716326 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716331 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716335 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716339 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716343 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716347 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716351 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716356 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716361 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716365 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716370 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716375 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716379 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716384 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716389 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716393 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716397 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716401 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:36.717273 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716406 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716410 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716414 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716423 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716429 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716433 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716438 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716443 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716448 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716452 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716457 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716461 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716465 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716470 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716474 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716478 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716483 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716487 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716492 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716496 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:36.718094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716500 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716505 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716509 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716513 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716518 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716524 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716528 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716533 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716537 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716542 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716546 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716551 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716555 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716559 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716564 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716569 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716573 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716577 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716581 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:36.718938 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716586 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716591 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716596 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716600 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716605 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716609 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716613 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716618 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716622 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716627 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716631 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716635 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716639 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716643 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716648 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716652 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716656 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716661 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716665 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716673 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:36.719875 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716677 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716682 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716686 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716690 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.716695 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.717986 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.717996 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718000 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718005 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718010 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718022 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718027 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718032 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718036 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718041 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718046 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718050 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718055 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718059 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:36.720433 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718064 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718068 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718073 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718078 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718082 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718086 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718090 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718094 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718098 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718105 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718110 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718116 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718121 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718126 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718130 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718135 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718139 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718143 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718148 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718152 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:36.720964 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718156 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718161 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718165 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718169 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718174 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718179 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718183 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718190 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718198 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718218 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718223 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718227 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718231 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718235 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718240 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718245 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718249 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718253 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718257 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718262 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:36.721480 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718266 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718270 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718274 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718279 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718284 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718290 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718295 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718299 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718303 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718307 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718311 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718315 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718320 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718324 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718328 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718333 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718337 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718342 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718346 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718351 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:36.721965 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718355 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718365 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718370 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718374 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718378 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718382 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718386 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718391 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718395 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718399 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718404 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718408 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718412 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718417 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718421 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718426 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718431 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718435 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718440 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:36.722795 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718444 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718449 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718453 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718457 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718462 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718466 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718470 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718474 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718478 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718483 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718489 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718493 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.718498 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719345 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719358 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719363 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719368 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719373 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719377 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719382 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:36.723617 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719386 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719390 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719395 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719402 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719407 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719413 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719417 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719422 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719427 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719433 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719442 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719448 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719453 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719458 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719462 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719466 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719471 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719475 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719479 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:36.724396 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719483 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719488 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719492 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719497 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719501 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719506 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719511 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719515 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719520 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719526 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719530 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719534 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719538 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719543 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719548 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719552 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719556 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719560 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719564 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719569 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:36.725118 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719573 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719578 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719582 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719586 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719590 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719595 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719599 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719603 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719607 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719612 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719616 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719621 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719625 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719630 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719634 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719638 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719642 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719649 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719654 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719658 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:36.725881 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719662 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719667 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719670 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719675 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719679 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719683 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719686 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719691 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719695 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719699 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719702 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719707 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719711 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719715 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719719 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719722 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719726 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719731 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719735 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719739 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:36.726650 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719743 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719747 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719751 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719755 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719759 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719763 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719767 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719772 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719776 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719782 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719786 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719792 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719797 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719802 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719807 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719811 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719816 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719820 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719825 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.719829 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:36.727148 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722079 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722104 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722116 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722124 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722131 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722136 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722143 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722150 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722155 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722160 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722166 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722171 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722177 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722182 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722187 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722192 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722196 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722215 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722222 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722227 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722232 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722237 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722244 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722251 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 22:00:36.727682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722256 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722262 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722267 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722273 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722278 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722287 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722294 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722299 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722306 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722311 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722316 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722321 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722326 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722330 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722338 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722344 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722349 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722353 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722358 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722365 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722370 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722375 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722380 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722385 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722390 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 22:00:36.728261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722395 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722400 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722404 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722409 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722414 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722423 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722429 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722434 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722439 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722444 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722449 2583 flags.go:64] FLAG: --help="false" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722455 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722461 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722466 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722471 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722477 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722483 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722488 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722492 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722497 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722502 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722507 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722513 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722518 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 22:00:36.728873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722523 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722527 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722532 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722537 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722541 2583 flags.go:64] FLAG: --lock-file="" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722546 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722551 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722557 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722566 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722571 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722576 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722581 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722585 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722593 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722598 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722603 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722610 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722615 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722621 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722626 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722632 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722637 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722642 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722648 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722653 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 22:00:36.729496 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722657 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722669 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722674 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722679 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722684 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722689 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722694 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722699 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722703 2583 flags.go:64] FLAG: --port="10250" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722708 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722713 2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-096dc96b296ec6c66" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722718 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722723 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722729 2583 flags.go:64] FLAG: --register-node="true" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722733 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722740 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722745 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722750 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722755 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722761 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722769 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722775 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722779 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722784 2583 flags.go:64] FLAG: --runonce="false" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722789 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722794 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722799 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 22:00:36.730098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722803 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722808 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722813 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722818 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722823 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722828 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722833 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722838 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722843 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722848 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722853 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722857 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722866 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722871 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722876 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722883 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722887 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722892 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722897 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722902 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722906 2583 flags.go:64] FLAG: --v="2" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722913 2583 flags.go:64] FLAG: --version="false" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722922 2583 flags.go:64] FLAG: --vmodule="" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722929 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.722934 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 22:00:36.730811 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723170 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723177 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723182 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723186 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723190 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723195 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723218 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723224 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723228 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723233 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723238 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723242 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723246 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723251 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723255 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723258 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723263 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723267 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723271 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723275 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:36.731428 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723280 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723284 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723289 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723293 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723297 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723301 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723306 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723310 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723314 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723321 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723325 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723329 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723334 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723338 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723342 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723347 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723351 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723356 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723360 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723364 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:36.731917 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723368 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723373 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723378 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723383 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723387 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723391 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723395 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723400 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723404 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723408 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723413 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723418 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723422 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723429 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723434 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723439 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723443 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723447 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723451 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:36.732422 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723456 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723462 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723470 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723476 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723481 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723486 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723491 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723495 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723499 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723504 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723508 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723513 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723517 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723522 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723527 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723531 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723537 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723541 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723545 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:36.732888 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723550 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723554 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723558 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723563 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723567 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723571 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.723664 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726186 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726198 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726216 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726223 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726292 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726443 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726449 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726452 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726455 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726460 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726462 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726467 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726471 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:36.733367 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726473 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726476 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726479 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726482 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726485 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726488 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726490 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726493 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.726530 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726597 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726605 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726610 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726613 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726616 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726619 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:36.733850 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726622 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726625 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726629 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726633 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726636 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726641 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726645 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726648 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726651 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726655 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726661 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726663 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726667 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726670 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726672 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726676 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726678 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726681 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726684 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:36.734198 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726687 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726689 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726692 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726695 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726698 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726700 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726703 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726706 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726708 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726711 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726714 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726717 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726719 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726722 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726724 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726727 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726730 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726732 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726735 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726738 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:36.734665 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726742 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726745 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726748 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726751 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726766 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726770 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726773 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726776 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726779 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726782 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726785 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726788 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726790 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726793 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726796 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726799 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726801 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726804 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726807 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726810 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:36.735137 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726812 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726815 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726818 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726821 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726824 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726827 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726830 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726832 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726835 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726837 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726840 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726843 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726846 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726851 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726853 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726856 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726859 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726861 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726864 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726867 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:36.735632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726869 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726872 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726875 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726877 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726880 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726883 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726885 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726888 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726897 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726900 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726902 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726905 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726908 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726911 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726913 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726916 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726919 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726921 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726924 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:36.736101 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726928 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.726930 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.733843 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.733861 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.733879 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.733884 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733939 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733946 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733949 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733952 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733956 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733958 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733961 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733964 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733967 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733970 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733972 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733975 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733978 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733981 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:36.736608 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733984 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733987 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733990 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733992 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733995 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.733998 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734001 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734003 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734006 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734008 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734011 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734014 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734016 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734019 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734021 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734026 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734029 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734032 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734035 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734038 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:36.737097 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734041 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734043 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734046 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734049 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734051 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734054 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734057 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734059 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734062 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734065 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734067 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734070 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734073 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734075 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734078 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734080 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734083 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734086 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734089 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:36.737595 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734092 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734094 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734097 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734099 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734102 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734105 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734108 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734111 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734114 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734117 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734120 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734123 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734125 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734129 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734134 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734137 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734140 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734143 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734145 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:36.738053 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734148 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734151 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734154 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734156 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734159 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734162 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734164 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734167 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734170 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734173 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734176 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734179 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734181 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734184 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734186 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734189 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734192 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734195 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734198 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734215 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:36.738597 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734219 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734222 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734225 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734228 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734231 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734234 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734239 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734242 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734246 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734248 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734251 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734254 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734257 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734259 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.734286 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:00:36.739088 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734425 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734430 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734434 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734437 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734440 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734442 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734445 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734448 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734450 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734453 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734456 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734459 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734461 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734465 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734467 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734470 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734473 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734476 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734478 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:36.739455 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734481 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734484 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734487 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734490 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734493 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734495 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734498 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734500 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734503 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734507 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734511 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734514 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734517 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734519 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734522 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734525 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734528 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734530 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734533 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734536 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:36.739925 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734538 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734541 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734544 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734546 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734549 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734552 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734555 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734557 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734560 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734563 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734566 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734569 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734573 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734576 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734579 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734583 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734585 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734588 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734591 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:36.740431 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734593 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734596 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734598 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734601 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734604 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734607 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734610 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734612 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734615 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734617 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734620 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734623 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734626 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734628 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734631 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734633 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734636 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734639 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734641 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734644 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:36.740860 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734646 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734649 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734651 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734654 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734657 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734660 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734662 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734665 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734668 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734671 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734674 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734677 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734679 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734682 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734684 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734687 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734690 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734693 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734695 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:36.741471 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734698 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734701 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734703 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734706 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734709 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734711 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734714 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734716 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734719 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.734742 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734772 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734776 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734779 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734782 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734785 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734788 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:00:36.741942 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734791 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734794 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734797 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734799 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734802 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734805 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734807 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734811 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734814 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734816 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734819 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734821 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734824 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734827 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734830 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734832 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734835 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734837 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734840 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734843 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:00:36.742334 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734845 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734848 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734850 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734853 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734856 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734859 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734862 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734864 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734867 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734870 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734873 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734875 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734878 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734880 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734883 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734887 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734890 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734892 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734895 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:00:36.742809 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734897 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734901 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734903 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734906 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734908 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734911 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734916 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734919 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734922 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734925 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734927 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734930 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734933 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734936 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734938 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734941 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734944 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734947 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734950 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:00:36.743289 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734952 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734955 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734958 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734961 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734965 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734968 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734971 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734975 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734978 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734981 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734984 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734987 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734989 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734992 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734995 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.734998 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735001 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735003 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735006 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:00:36.743741 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735010 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735013 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735015 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735018 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735021 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735023 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735026 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735029 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735034 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735038 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735041 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735044 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735047 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735049 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735052 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735055 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735057 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735060 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735063 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:00:36.744189 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735065 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735068 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735071 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:36.735073 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.735132 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.737289 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.738431 2583 server.go:1014] "Starting client certificate rotation" Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.738552 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.738584 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.740539 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 22:00:36.744683 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.743830 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 22:00:36.752439 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.752423 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 22:00:36.756033 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.756018 2583 log.go:25] "Validated CRI v1 image API" Oct 30 22:00:36.757888 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.757688 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 22:00:36.763883 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.763861 2583 fs.go:135] Filesystem UUIDs: map[24c5e27f-3136-491c-8c5c-45ce9470f807:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f04bdf9d-7ee1-42a3-adeb-ddb1bfa1499d:/dev/nvme0n1p4] Oct 30 22:00:36.763970 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.763882 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 22:00:36.769430 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.769311 2583 manager.go:217] Machine: {Timestamp:2026-10-30 22:00:36.76771558 +0000 UTC m=+0.355790364 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099823 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c3ab4f3fbce05a8e4803debe0cffe SystemUUID:ec2c3ab4-f3fb-ce05-a8e4-803debe0cffe BootID:fd8771b6-8815-4c05-8996-e69a1525746d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d3:bd:9f:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d3:bd:9f:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:2b:51:d1:3c:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 22:00:36.769926 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.769912 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 22:00:36.770073 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.770057 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 22:00:36.771282 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.771257 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 22:00:36.771419 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.771284 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-194.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 22:00:36.771462 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.771422 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 22:00:36.771462 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.771429 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 22:00:36.771519 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.771472 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 22:00:36.771519 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.771490 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 22:00:36.772445 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.772435 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 22:00:36.775634 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.775623 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 22:00:36.775755 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.775746 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 22:00:36.776493 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.776484 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 22:00:36.776532 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.776497 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 22:00:36.776532 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.776507 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 22:00:36.776532 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.776516 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 22:00:36.776532 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.776524 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 22:00:36.776850 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.776839 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 22:00:36.779453 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.779435 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 22:00:36.785687 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.785672 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 22:00:36.786662 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.786649 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 22:00:36.786708 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.786679 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 22:00:36.786847 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.786835 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 22:00:36.786887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.786856 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 22:00:36.786887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.786867 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 22:00:36.786887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.786875 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 22:00:36.786887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.786882 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 22:00:36.786887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.786890 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 22:00:36.787641 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.787631 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 22:00:36.787670 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.787641 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 22:00:36.787670 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.787650 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 22:00:36.787670 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.787663 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 22:00:36.787749 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.787675 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 22:00:36.787749 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.787685 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 22:00:36.788472 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.788464 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 22:00:36.788472 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.788473 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 22:00:36.792240 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.792227 2583 server.go:1263] "Started kubelet" Oct 30 22:00:36.792442 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.792388 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 22:00:36.792523 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.792511 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 22:00:36.792605 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.792416 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 22:00:36.793000 ip-10-0-141-194 systemd[1]: Started Kubernetes Kubelet. Oct 30 22:00:36.793835 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.793759 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 22:00:36.796602 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.796586 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 22:00:36.799410 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.799393 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 22:00:36.801756 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.801737 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 22:00:36.802707 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.802689 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 22:00:36.803680 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.803643 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" Oct 30 22:00:36.803840 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.803824 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 22:00:36.803840 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.803842 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 22:00:36.803977 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.803960 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 22:00:36.804161 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.804142 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 22:00:36.804161 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.804159 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 22:00:36.805987 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.805811 2583 factory.go:55] Registering systemd factory Oct 30 22:00:36.805987 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.805874 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 22:00:36.806548 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.806526 2583 factory.go:153] Registering CRI-O factory Oct 30 22:00:36.806606 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.806554 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 22:00:36.806662 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.806610 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 22:00:36.806662 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.806629 2583 factory.go:103] Registering Raw factory Oct 30 22:00:36.806662 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.806647 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 22:00:36.807602 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.807585 2583 manager.go:319] Starting recovery of all containers Oct 30 22:00:36.813635 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.813588 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 22:00:36.817985 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.817966 2583 manager.go:324] Recovery completed Oct 30 22:00:36.822566 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.822554 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:36.824063 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.824046 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:36.824124 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.824082 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:36.824124 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.824098 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:36.824763 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.824746 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 22:00:36.824763 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.824758 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 22:00:36.824889 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.824776 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 22:00:36.824889 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.824771 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:36.825648 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.825629 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:36.825736 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.825662 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:36.825736 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.825676 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:36.826141 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.826124 2583 policy_none.go:50] "Start" Oct 30 22:00:36.826141 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.826141 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 22:00:36.826299 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.826152 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 22:00:36.828025 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.828014 2583 policy_none.go:44] "Start" Oct 30 22:00:36.855090 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.855031 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 22:00:36.860376 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.860359 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 22:00:36.860502 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.860394 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 22:00:36.860502 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.860406 2583 server.go:87] "Starting device plugin registration server" Oct 30 22:00:36.860668 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.860656 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 22:00:36.860701 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.860671 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 22:00:36.860811 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.860794 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 22:00:36.861152 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.860880 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 22:00:36.861152 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.860889 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 22:00:36.861482 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.861464 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 22:00:36.861558 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.861509 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" Oct 30 22:00:36.928407 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.928380 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-194.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 22:00:36.930109 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.928390 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-194.us-west-2.compute.internal.18e36da61a05f0c2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-194.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:36.792185026 +0000 UTC m=+0.380259811,LastTimestamp:2026-10-30 22:00:36.792185026 +0000 UTC m=+0.380259811,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-194.us-west-2.compute.internal,}" Oct 30 22:00:36.952451 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.952426 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 22:00:36.952451 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.952455 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 22:00:36.952611 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.952479 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 22:00:36.952611 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.952516 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 22:00:36.962578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.962506 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:36.966133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.966111 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:36.966273 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.966148 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:36.966273 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.966164 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:36.966273 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:36.966189 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:36.973224 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.973189 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-194.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 22:00:36.977022 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:36.976947 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-194.us-west-2.compute.internal.18e36da61bec63cd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-194.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-194.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:36.824064973 +0000 UTC m=+0.412139758,LastTimestamp:2026-10-30 22:00:36.824064973 +0000 UTC m=+0.412139758,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-194.us-west-2.compute.internal,}" Oct 30 22:00:37.029993 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.029955 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 22:00:37.030174 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.030037 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ptx4l" Oct 30 22:00:37.052625 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.052589 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal"] Oct 30 22:00:37.052794 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.052661 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:37.053689 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.053662 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:37.053789 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.053700 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:37.053789 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.053714 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:37.055188 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.055172 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:37.055848 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.055833 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.055937 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.055869 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:37.055937 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.055915 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:37.056024 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.055945 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:37.056024 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.055956 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:37.056557 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.056541 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:37.056620 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.056568 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:37.056620 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.056578 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:37.057388 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.057375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.057458 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.057400 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:00:37.058106 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.058085 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:00:37.058216 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.058113 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:00:37.058216 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.058126 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:00:37.091845 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.091598 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:37.094708 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.094687 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:37.096625 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:37.096605 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.098254 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:37.098236 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.105364 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.105345 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.105443 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.105371 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.105443 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.105388 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7419d42fafb37e8b2d87d41a6e268c81-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"7419d42fafb37e8b2d87d41a6e268c81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.185027 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.184997 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:37.185696 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.185675 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:37.187087 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.187069 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.204559 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.204536 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.205675 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.205650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.205777 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.205673 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.205777 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.205694 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.205777 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.205718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7419d42fafb37e8b2d87d41a6e268c81-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"7419d42fafb37e8b2d87d41a6e268c81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.205777 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.205758 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7419d42fafb37e8b2d87d41a6e268c81-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"7419d42fafb37e8b2d87d41a6e268c81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.205918 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.205793 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.290671 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.290597 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:00:37.290805 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.290728 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.350514 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.350484 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:00:37.400707 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.400670 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.402817 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.402799 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:00:37.739305 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.739022 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 22:00:37.739891 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.739403 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 22:00:37.739891 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.739458 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 22:00:37.739891 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.739482 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 22:00:37.739891 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:37.739454 2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a56facd814ea6491ea434b7fb55cee90-852268fd564bd9ac.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-194.us-west-2.compute.internal.18e36da61becfec5\": read tcp 10.0.141.194:48970->3.16.100.64:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-194.us-west-2.compute.internal.18e36da61becfec5 default 5145 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-194.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-194.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-194.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 22:00:36 +0000 UTC,LastTimestamp:2026-10-30 22:00:36.966168643 +0000 UTC m=+0.554243426,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-194.us-west-2.compute.internal,}" Oct 30 22:00:37.778035 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.777997 2583 apiserver.go:52] "Watching apiserver" Oct 30 22:00:37.802221 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:37.802167 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 22:00:38.035804 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.035706 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:55:37 +0000 UTC" deadline="2028-08-13 17:24:39.15862625 +0000 UTC" Oct 30 22:00:38.035804 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.035747 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15667h24m1.122883599s" Oct 30 22:00:38.062069 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.062033 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 22:00:38.062503 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.062475 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-2qtcg","kube-system/global-pull-secret-syncer-h8pdj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2","openshift-ovn-kubernetes/ovnkube-node-mlgss","kube-system/konnectivity-agent-4k4pr","openshift-image-registry/node-ca-47nqg","openshift-cluster-node-tuning-operator/tuned-c44r7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal","openshift-multus/multus-xc24q","kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal","openshift-dns/node-resolver-rg6kq","openshift-multus/multus-additional-cni-plugins-trblz","openshift-multus/network-metrics-daemon-5qc57","openshift-network-diagnostics/network-check-target-6mdvj"] Oct 30 22:00:38.064145 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.064122 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.065542 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.065522 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.065645 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.065603 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:38.066793 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.066777 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.068418 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.068394 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.070526 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.070492 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:38.072051 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.072033 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.073499 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.073485 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.074906 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.074889 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.076456 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.076434 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.077807 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.077792 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.079203 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.079188 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:38.079297 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.079263 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:38.080514 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.080497 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:38.080567 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.080553 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:38.105562 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.105537 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 22:00:38.110432 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110408 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-tmp\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.110542 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110439 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-hostroot\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.110542 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110459 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-multus-certs\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.110542 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110483 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-systemd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.110651 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-env-overrides\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.110651 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-modprobe-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.110651 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.110748 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110662 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.110748 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110681 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-netns\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.110748 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110698 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.110748 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110737 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.110869 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110776 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-system-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.110869 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-multus\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.110869 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110832 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-etc-kubernetes\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.110869 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110846 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysconfig\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.110869 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110860 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-dbus\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.111018 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110897 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cni-binary-copy\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.111018 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.111018 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.110972 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-kubernetes\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.111018 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-systemd\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.111176 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-ovn\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111176 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-host\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.111176 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-binary-copy\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.111176 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111105 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-slash\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111176 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111176 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111158 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a72d78e-c521-4dba-b551-5970a7505245-ovn-node-metrics-cert\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111440 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ca8590-27e5-497d-9bb8-cb5041ceb267-host-slash\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.111440 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111198 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20b0ee9f-f99d-472a-a1d8-df44248d9aec-konnectivity-ca\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:38.111440 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-etc-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111440 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111349 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-bin\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111440 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-run\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.111646 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111450 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-var-lib-kubelet\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.111646 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111468 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-device-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.111646 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111503 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfzp9\" (UniqueName: \"kubernetes.io/projected/c986bec8-11a1-4c93-86fa-40b486199598-kube-api-access-vfzp9\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.111646 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111530 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-socket-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.111646 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111546 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-netns\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.111646 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111589 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111646 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111636 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-socket-dir-parent\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.111896 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111657 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-var-lib-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111896 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111685 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cnibin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.111896 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111733 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/94ca8590-27e5-497d-9bb8-cb5041ceb267-iptables-alerter-script\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.111896 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111780 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-sys-fs\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.111896 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111811 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-netd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.111896 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111834 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-lib-modules\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.111896 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111894 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c986bec8-11a1-4c93-86fa-40b486199598-hosts-file\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111911 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jq7\" (UniqueName: \"kubernetes.io/projected/94ca8590-27e5-497d-9bb8-cb5041ceb267-kube-api-access-g8jq7\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111926 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20b0ee9f-f99d-472a-a1d8-df44248d9aec-agent-certs\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.111981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-systemd-units\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112007 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpgg\" (UniqueName: \"kubernetes.io/projected/1a72d78e-c521-4dba-b551-5970a7505245-kube-api-access-5gpgg\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-conf\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112087 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445ts\" (UniqueName: \"kubernetes.io/projected/8f434211-8bee-4801-9c8e-39f0bb0e71d6-kube-api-access-445ts\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.112118 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-kubelet\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112128 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-node-log\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcv94\" (UniqueName: \"kubernetes.io/projected/cfbc34e7-48ae-456a-ad03-408b46e20053-kube-api-access-hcv94\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-conf-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112286 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-daemon-config\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112325 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-sys\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-etc-tuned\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112371 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd8k8\" (UniqueName: \"kubernetes.io/projected/8420c80f-804e-468e-8df2-fb1ec145d318-kube-api-access-dd8k8\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112404 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112444 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-kubelet\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-system-cni-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112522 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112554 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c986bec8-11a1-4c93-86fa-40b486199598-tmp-dir\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-kubelet-config\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kxtv\" (UniqueName: \"kubernetes.io/projected/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kube-api-access-8kxtv\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112656 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-k8s-cni-cncf-io\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112685 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-bin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112710 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-log-socket\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.112723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112725 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-config\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.113013 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b54602-8b76-49b7-b295-42969da3c4fd-host\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.113013 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112776 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:38.113013 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-registration-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.113013 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-cnibin\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.113013 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112854 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6frbn\" (UniqueName: \"kubernetes.io/projected/1b8fc6dd-36fe-4714-bcda-1863280a8324-kube-api-access-6frbn\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.113013 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-script-lib\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.113013 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112904 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3b54602-8b76-49b7-b295-42969da3c4fd-serviceca\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.113013 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.112938 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkwvd\" (UniqueName: \"kubernetes.io/projected/b3b54602-8b76-49b7-b295-42969da3c4fd-kube-api-access-nkwvd\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.131576 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.131549 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 22:00:38.131695 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.131622 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 22:00:38.131695 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.131648 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 22:00:38.131787 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.131700 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75hcr\"" Oct 30 22:00:38.131787 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.131779 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dj92j\"" Oct 30 22:00:38.131864 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.131781 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xd6xx\"" Oct 30 22:00:38.131864 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.131788 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skckn\"" Oct 30 22:00:38.134737 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134715 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134740 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134802 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134826 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134910 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134917 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134927 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134950 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134970 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 22:00:38.134973 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.134980 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 22:00:38.135367 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.135243 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 22:00:38.185226 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.185202 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 22:00:38.185361 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.185275 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nqw7z" Oct 30 22:00:38.214045 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.213997 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c986bec8-11a1-4c93-86fa-40b486199598-tmp-dir\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.214045 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-kubelet-config\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kxtv\" (UniqueName: \"kubernetes.io/projected/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kube-api-access-8kxtv\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214090 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-k8s-cni-cncf-io\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-bin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214136 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-log-socket\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214144 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-kubelet-config\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214163 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-config\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b54602-8b76-49b7-b295-42969da3c4fd-host\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214238 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-k8s-cni-cncf-io\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214259 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-bin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.214272 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214246 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-log-socket\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214290 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-registration-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214300 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b54602-8b76-49b7-b295-42969da3c4fd-host\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214360 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c986bec8-11a1-4c93-86fa-40b486199598-tmp-dir\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214408 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-registration-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214404 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-cnibin\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6frbn\" (UniqueName: \"kubernetes.io/projected/1b8fc6dd-36fe-4714-bcda-1863280a8324-kube-api-access-6frbn\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214470 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-cnibin\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.214619 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-script-lib\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3b54602-8b76-49b7-b295-42969da3c4fd-serviceca\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214654 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwvd\" (UniqueName: \"kubernetes.io/projected/b3b54602-8b76-49b7-b295-42969da3c4fd-kube-api-access-nkwvd\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-tmp\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214747 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-hostroot\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214750 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-config\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-multus-certs\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-systemd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214829 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-env-overrides\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214829 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-hostroot\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-modprobe-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214879 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-multus-certs\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214900 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-systemd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.214975 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214954 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.214992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-netns\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215003 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-modprobe-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215027 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215049 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215065 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-netns\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215074 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-system-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215096 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-multus\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-etc-kubernetes\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysconfig\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.215159 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215164 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-multus\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215176 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-dbus\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215178 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-system-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215126 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-script-lib\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215202 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-etc-kubernetes\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.215578 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215220 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysconfig\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.215226 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret podName:2675c1b0-71d1-40b8-b78c-5ef4a59389d7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:38.715205329 +0000 UTC m=+2.303280117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret") pod "global-pull-secret-syncer-h8pdj" (UID: "2675c1b0-71d1-40b8-b78c-5ef4a59389d7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215156 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cni-binary-copy\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215246 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-env-overrides\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215296 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-dbus\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215293 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-kubernetes\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-systemd\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215388 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-ovn\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215390 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-kubernetes\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215426 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-ovn\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215425 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-host\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-binary-copy\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215462 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-systemd\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-slash\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215510 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-host\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.216497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215512 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-slash\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215554 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215576 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a72d78e-c521-4dba-b551-5970a7505245-ovn-node-metrics-cert\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ca8590-27e5-497d-9bb8-cb5041ceb267-host-slash\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215626 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20b0ee9f-f99d-472a-a1d8-df44248d9aec-konnectivity-ca\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-etc-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215679 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ca8590-27e5-497d-9bb8-cb5041ceb267-host-slash\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215682 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-bin\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-run\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215732 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-etc-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-var-lib-kubelet\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215782 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-bin\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215801 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-var-lib-kubelet\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215812 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-device-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215839 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzp9\" (UniqueName: \"kubernetes.io/projected/c986bec8-11a1-4c93-86fa-40b486199598-kube-api-access-vfzp9\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215819 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-run\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.217379 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215881 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-device-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-socket-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-netns\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.215984 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216046 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216056 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-socket-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216055 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-netns\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216086 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-socket-dir-parent\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216129 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-var-lib-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216158 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cnibin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216185 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-socket-dir-parent\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216189 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-var-lib-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/94ca8590-27e5-497d-9bb8-cb5041ceb267-iptables-alerter-script\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216260 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20b0ee9f-f99d-472a-a1d8-df44248d9aec-konnectivity-ca\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216262 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cnibin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-sys-fs\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216324 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-netd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-lib-modules\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.217887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216353 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-sys-fs\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c986bec8-11a1-4c93-86fa-40b486199598-hosts-file\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216372 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-netd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216388 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jq7\" (UniqueName: \"kubernetes.io/projected/94ca8590-27e5-497d-9bb8-cb5041ceb267-kube-api-access-g8jq7\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216412 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216432 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c986bec8-11a1-4c93-86fa-40b486199598-hosts-file\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216447 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20b0ee9f-f99d-472a-a1d8-df44248d9aec-agent-certs\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216451 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-lib-modules\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-systemd-units\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216500 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpgg\" (UniqueName: \"kubernetes.io/projected/1a72d78e-c521-4dba-b551-5970a7505245-kube-api-access-5gpgg\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216516 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216525 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-conf\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216566 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-systemd-units\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216579 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445ts\" (UniqueName: \"kubernetes.io/projected/8f434211-8bee-4801-9c8e-39f0bb0e71d6-kube-api-access-445ts\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-kubelet\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-node-log\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.218422 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216688 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-conf\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcv94\" (UniqueName: \"kubernetes.io/projected/cfbc34e7-48ae-456a-ad03-408b46e20053-kube-api-access-hcv94\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-conf-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216767 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/94ca8590-27e5-497d-9bb8-cb5041ceb267-iptables-alerter-script\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216779 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-daemon-config\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.216786 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216820 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-sys\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216851 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-etc-tuned\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216853 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.216887 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs podName:cfbc34e7-48ae-456a-ad03-408b46e20053 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:38.716863643 +0000 UTC m=+2.304938627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs") pod "network-metrics-daemon-5qc57" (UID: "cfbc34e7-48ae-456a-ad03-408b46e20053") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216899 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-conf-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216910 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8k8\" (UniqueName: \"kubernetes.io/projected/8420c80f-804e-468e-8df2-fb1ec145d318-kube-api-access-dd8k8\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216931 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-sys\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-kubelet\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216994 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-system-cni-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.219133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.217031 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.217069 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216852 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-kubelet\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.217169 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-kubelet\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.216993 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-node-log\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.217235 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-system-cni-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.219143 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-tmp\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.219160 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a72d78e-c521-4dba-b551-5970a7505245-ovn-node-metrics-cert\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.219194 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-etc-tuned\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.220029 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.219272 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20b0ee9f-f99d-472a-a1d8-df44248d9aec-agent-certs\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:38.240682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.240643 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gzw7r\"" Oct 30 22:00:38.240682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.240646 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdfn5\"" Oct 30 22:00:38.240682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.240646 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6qlsl\"" Oct 30 22:00:38.240682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.240654 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hlw52\"" Oct 30 22:00:38.240938 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.240697 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfj4m\"" Oct 30 22:00:38.243881 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.243864 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 22:00:38.244028 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.243972 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.244028 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.243980 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 22:00:38.244028 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244000 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.244028 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244002 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 22:00:38.244028 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244022 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.244265 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244041 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 22:00:38.244265 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244106 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.244265 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244111 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.244265 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244115 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.244265 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244131 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 22:00:38.244265 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244142 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 22:00:38.244265 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.244143 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 22:00:38.246497 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.246479 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3b54602-8b76-49b7-b295-42969da3c4fd-serviceca\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.246580 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.246564 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cni-binary-copy\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.246623 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.246563 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.247385 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.247368 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-binary-copy\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.247734 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.247718 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.247814 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.247798 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-daemon-config\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.277939 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.277833 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:38.277939 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.277861 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:38.277939 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.277893 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gzgj8 for pod openshift-network-diagnostics/network-check-target-6mdvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:38.278274 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.277980 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8 podName:588b62a8-9745-4e5a-97dd-f8c451197018 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:38.777943969 +0000 UTC m=+2.366018749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gzgj8" (UniqueName: "kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8") pod "network-check-target-6mdvj" (UID: "588b62a8-9745-4e5a-97dd-f8c451197018") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:38.279738 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.279706 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwvd\" (UniqueName: \"kubernetes.io/projected/b3b54602-8b76-49b7-b295-42969da3c4fd-kube-api-access-nkwvd\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.280180 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.280150 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzp9\" (UniqueName: \"kubernetes.io/projected/c986bec8-11a1-4c93-86fa-40b486199598-kube-api-access-vfzp9\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.280367 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.280348 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jq7\" (UniqueName: \"kubernetes.io/projected/94ca8590-27e5-497d-9bb8-cb5041ceb267-kube-api-access-g8jq7\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.280699 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.280672 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kxtv\" (UniqueName: \"kubernetes.io/projected/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kube-api-access-8kxtv\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.280887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.280869 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpgg\" (UniqueName: \"kubernetes.io/projected/1a72d78e-c521-4dba-b551-5970a7505245-kube-api-access-5gpgg\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.281293 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.281271 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6frbn\" (UniqueName: \"kubernetes.io/projected/1b8fc6dd-36fe-4714-bcda-1863280a8324-kube-api-access-6frbn\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.282057 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.282033 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445ts\" (UniqueName: \"kubernetes.io/projected/8f434211-8bee-4801-9c8e-39f0bb0e71d6-kube-api-access-445ts\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.284703 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.284675 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcv94\" (UniqueName: \"kubernetes.io/projected/cfbc34e7-48ae-456a-ad03-408b46e20053-kube-api-access-hcv94\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:38.286022 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.285978 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8k8\" (UniqueName: \"kubernetes.io/projected/8420c80f-804e-468e-8df2-fb1ec145d318-kube-api-access-dd8k8\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.392299 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.392261 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:00:38.413015 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.412984 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:00:38.435866 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.435828 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:00:38.447154 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.447127 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:38.450135 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.450117 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:00:38.475099 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.475067 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:00:38.484027 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.484003 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xc24q" Oct 30 22:00:38.495511 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.495489 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:00:38.506965 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.506941 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:00:38.645025 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:38.644991 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7419d42fafb37e8b2d87d41a6e268c81.slice/crio-4642ae4eba01be1299c42eb440d862b9af8b1a2089e6d7b0f1ada23dedd6c357 WatchSource:0}: Error finding container 4642ae4eba01be1299c42eb440d862b9af8b1a2089e6d7b0f1ada23dedd6c357: Status 404 returned error can't find the container with id 4642ae4eba01be1299c42eb440d862b9af8b1a2089e6d7b0f1ada23dedd6c357 Oct 30 22:00:38.645632 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:38.645604 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d00c89f3615755885c52b7effe8bce.slice/crio-0cfbc5e5215ff32bb6d5854f5dcd92f3efc54b9be9172cde3ab1344e2176025c WatchSource:0}: Error finding container 0cfbc5e5215ff32bb6d5854f5dcd92f3efc54b9be9172cde3ab1344e2176025c: Status 404 returned error can't find the container with id 0cfbc5e5215ff32bb6d5854f5dcd92f3efc54b9be9172cde3ab1344e2176025c Oct 30 22:00:38.652329 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.652308 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:00:38.720557 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.720523 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:38.720719 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.720574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:38.720719 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.720693 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:38.720790 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.720754 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs podName:cfbc34e7-48ae-456a-ad03-408b46e20053 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:39.720738855 +0000 UTC m=+3.308813628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs") pod "network-metrics-daemon-5qc57" (UID: "cfbc34e7-48ae-456a-ad03-408b46e20053") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:38.720790 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.720693 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:38.720866 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.720797 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret podName:2675c1b0-71d1-40b8-b78c-5ef4a59389d7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:39.720790603 +0000 UTC m=+3.308865376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret") pod "global-pull-secret-syncer-h8pdj" (UID: "2675c1b0-71d1-40b8-b78c-5ef4a59389d7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:38.821198 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.821151 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:38.821795 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.821351 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:38.821795 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.821375 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:38.821795 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.821390 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gzgj8 for pod openshift-network-diagnostics/network-check-target-6mdvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:38.821795 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:38.821441 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8 podName:588b62a8-9745-4e5a-97dd-f8c451197018 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:39.82142643 +0000 UTC m=+3.409501203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzgj8" (UniqueName: "kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8") pod "network-check-target-6mdvj" (UID: "588b62a8-9745-4e5a-97dd-f8c451197018") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:38.956883 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.956824 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"7419d42fafb37e8b2d87d41a6e268c81","Type":"ContainerStarted","Data":"4642ae4eba01be1299c42eb440d862b9af8b1a2089e6d7b0f1ada23dedd6c357"} Oct 30 22:00:38.957936 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:38.957913 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"56d00c89f3615755885c52b7effe8bce","Type":"ContainerStarted","Data":"0cfbc5e5215ff32bb6d5854f5dcd92f3efc54b9be9172cde3ab1344e2176025c"} Oct 30 22:00:39.188095 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.188030 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:38 +0000 UTC" deadline="2028-05-25 23:21:07.148671519 +0000 UTC" Oct 30 22:00:39.188095 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.188082 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13753h20m27.960594167s" Oct 30 22:00:39.311055 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.310963 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:39.312063 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.312043 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:39.329792 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.329761 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:00:39.355410 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:39.355380 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc986bec8_11a1_4c93_86fa_40b486199598.slice/crio-af1c410213e019092d196caca5c15b78b8f9190741c49bc0f14d0b3a18dc9ee5 WatchSource:0}: Error finding container af1c410213e019092d196caca5c15b78b8f9190741c49bc0f14d0b3a18dc9ee5: Status 404 returned error can't find the container with id af1c410213e019092d196caca5c15b78b8f9190741c49bc0f14d0b3a18dc9ee5 Oct 30 22:00:39.429494 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:39.429465 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adebb4b_a9a3_4c8e_89c2_940cfcab9e28.slice/crio-4f7fd4d0a8e58793dd57e2366d805e66f5610b2adebbcc7a33cc1e6af6779994 WatchSource:0}: Error finding container 4f7fd4d0a8e58793dd57e2366d805e66f5610b2adebbcc7a33cc1e6af6779994: Status 404 returned error can't find the container with id 4f7fd4d0a8e58793dd57e2366d805e66f5610b2adebbcc7a33cc1e6af6779994 Oct 30 22:00:39.444413 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:39.444375 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b54602_8b76_49b7_b295_42969da3c4fd.slice/crio-0679eeb4ff57c90459211f3d32808d284ca564b9d34c24c8e09552cd4e2007d5 WatchSource:0}: Error finding container 0679eeb4ff57c90459211f3d32808d284ca564b9d34c24c8e09552cd4e2007d5: Status 404 returned error can't find the container with id 0679eeb4ff57c90459211f3d32808d284ca564b9d34c24c8e09552cd4e2007d5 Oct 30 22:00:39.470326 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:39.470272 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b0ee9f_f99d_472a_a1d8_df44248d9aec.slice/crio-2c319c110faae66e896cb9bb8e246decbe483c4857bbc3e6f6ea12e5916b9af7 WatchSource:0}: Error finding container 2c319c110faae66e896cb9bb8e246decbe483c4857bbc3e6f6ea12e5916b9af7: Status 404 returned error can't find the container with id 2c319c110faae66e896cb9bb8e246decbe483c4857bbc3e6f6ea12e5916b9af7 Oct 30 22:00:39.483951 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:39.483925 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ca8590_27e5_497d_9bb8_cb5041ceb267.slice/crio-d4068e99b303b1e9e7c590611c43a0fdcb2c0e02998f52454ec7d1e61417f808 WatchSource:0}: Error finding container d4068e99b303b1e9e7c590611c43a0fdcb2c0e02998f52454ec7d1e61417f808: Status 404 returned error can't find the container with id d4068e99b303b1e9e7c590611c43a0fdcb2c0e02998f52454ec7d1e61417f808 Oct 30 22:00:39.493745 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:39.493707 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a72d78e_c521_4dba_b551_5970a7505245.slice/crio-22e292e20416440ade309fdfa2e737c5af722d3d10d76f629bb6bfd3b07df45f WatchSource:0}: Error finding container 22e292e20416440ade309fdfa2e737c5af722d3d10d76f629bb6bfd3b07df45f: Status 404 returned error can't find the container with id 22e292e20416440ade309fdfa2e737c5af722d3d10d76f629bb6bfd3b07df45f Oct 30 22:00:39.503892 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:39.503867 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f434211_8bee_4801_9c8e_39f0bb0e71d6.slice/crio-41c1ffe91871d12bf833318f37eab174b36f86bf4002ebe9dd45ae4fa1d24983 WatchSource:0}: Error finding container 41c1ffe91871d12bf833318f37eab174b36f86bf4002ebe9dd45ae4fa1d24983: Status 404 returned error can't find the container with id 41c1ffe91871d12bf833318f37eab174b36f86bf4002ebe9dd45ae4fa1d24983 Oct 30 22:00:39.507094 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:00:39.507074 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8420c80f_804e_468e_8df2_fb1ec145d318.slice/crio-6594f73aba8e89cca3ffc83f3e0d0ed4525cd3335dead614bb23d9e67e090066 WatchSource:0}: Error finding container 6594f73aba8e89cca3ffc83f3e0d0ed4525cd3335dead614bb23d9e67e090066: Status 404 returned error can't find the container with id 6594f73aba8e89cca3ffc83f3e0d0ed4525cd3335dead614bb23d9e67e090066 Oct 30 22:00:39.728265 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.728211 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:39.728439 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.728277 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:39.728439 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.728368 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:39.728439 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.728370 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:39.728439 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.728419 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs podName:cfbc34e7-48ae-456a-ad03-408b46e20053 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:41.728405308 +0000 UTC m=+5.316480096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs") pod "network-metrics-daemon-5qc57" (UID: "cfbc34e7-48ae-456a-ad03-408b46e20053") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:39.728439 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.728433 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret podName:2675c1b0-71d1-40b8-b78c-5ef4a59389d7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:41.7284269 +0000 UTC m=+5.316501672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret") pod "global-pull-secret-syncer-h8pdj" (UID: "2675c1b0-71d1-40b8-b78c-5ef4a59389d7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:39.828819 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.828785 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:39.829462 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.828981 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:39.829462 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.829001 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:39.829462 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.829011 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gzgj8 for pod openshift-network-diagnostics/network-check-target-6mdvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:39.829462 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.829075 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8 podName:588b62a8-9745-4e5a-97dd-f8c451197018 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:41.829054459 +0000 UTC m=+5.417129235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzgj8" (UniqueName: "kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8") pod "network-check-target-6mdvj" (UID: "588b62a8-9745-4e5a-97dd-f8c451197018") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:39.953047 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.953012 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:39.953251 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.953092 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:39.953251 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.953194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:39.953251 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.953227 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:39.953433 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.953317 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:39.953491 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:39.953433 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:39.961833 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.961778 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qtcg" event={"ID":"94ca8590-27e5-497d-9bb8-cb5041ceb267","Type":"ContainerStarted","Data":"d4068e99b303b1e9e7c590611c43a0fdcb2c0e02998f52454ec7d1e61417f808"} Oct 30 22:00:39.963206 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.963178 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-47nqg" event={"ID":"b3b54602-8b76-49b7-b295-42969da3c4fd","Type":"ContainerStarted","Data":"0679eeb4ff57c90459211f3d32808d284ca564b9d34c24c8e09552cd4e2007d5"} Oct 30 22:00:39.964715 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.964686 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" event={"ID":"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28","Type":"ContainerStarted","Data":"4f7fd4d0a8e58793dd57e2366d805e66f5610b2adebbcc7a33cc1e6af6779994"} Oct 30 22:00:39.965967 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.965929 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rg6kq" event={"ID":"c986bec8-11a1-4c93-86fa-40b486199598","Type":"ContainerStarted","Data":"af1c410213e019092d196caca5c15b78b8f9190741c49bc0f14d0b3a18dc9ee5"} Oct 30 22:00:39.967392 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.967358 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c44r7" event={"ID":"8420c80f-804e-468e-8df2-fb1ec145d318","Type":"ContainerStarted","Data":"6594f73aba8e89cca3ffc83f3e0d0ed4525cd3335dead614bb23d9e67e090066"} Oct 30 22:00:39.968873 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.968844 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"22e292e20416440ade309fdfa2e737c5af722d3d10d76f629bb6bfd3b07df45f"} Oct 30 22:00:39.970222 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.970195 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4k4pr" event={"ID":"20b0ee9f-f99d-472a-a1d8-df44248d9aec","Type":"ContainerStarted","Data":"2c319c110faae66e896cb9bb8e246decbe483c4857bbc3e6f6ea12e5916b9af7"} Oct 30 22:00:39.971513 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.971475 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trblz" event={"ID":"1b8fc6dd-36fe-4714-bcda-1863280a8324","Type":"ContainerStarted","Data":"c768f6881cdb1936971434cd7c791005a8928c6e83cbac8b7bfd9bda816289aa"} Oct 30 22:00:39.972723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:39.972696 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xc24q" event={"ID":"8f434211-8bee-4801-9c8e-39f0bb0e71d6","Type":"ContainerStarted","Data":"41c1ffe91871d12bf833318f37eab174b36f86bf4002ebe9dd45ae4fa1d24983"} Oct 30 22:00:40.188989 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:40.188920 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:38 +0000 UTC" deadline="2028-04-25 13:21:41.199408896 +0000 UTC" Oct 30 22:00:40.188989 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:40.188984 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13023h21m1.010429435s" Oct 30 22:00:41.755796 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:41.755351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:41.755796 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:41.755434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:41.755796 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.755592 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:41.755796 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.755657 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs podName:cfbc34e7-48ae-456a-ad03-408b46e20053 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:45.755637855 +0000 UTC m=+9.343712634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs") pod "network-metrics-daemon-5qc57" (UID: "cfbc34e7-48ae-456a-ad03-408b46e20053") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:41.755796 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.755725 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:41.755796 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.755762 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret podName:2675c1b0-71d1-40b8-b78c-5ef4a59389d7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:45.755750681 +0000 UTC m=+9.343825457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret") pod "global-pull-secret-syncer-h8pdj" (UID: "2675c1b0-71d1-40b8-b78c-5ef4a59389d7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:41.857175 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:41.856740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:41.857175 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.856948 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:41.857175 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.856969 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:41.857175 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.856982 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gzgj8 for pod openshift-network-diagnostics/network-check-target-6mdvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:41.857175 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.857045 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8 podName:588b62a8-9745-4e5a-97dd-f8c451197018 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:45.857026401 +0000 UTC m=+9.445101183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzgj8" (UniqueName: "kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8") pod "network-check-target-6mdvj" (UID: "588b62a8-9745-4e5a-97dd-f8c451197018") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:41.953299 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:41.953075 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:41.953299 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:41.953107 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:41.953299 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.953243 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:41.953631 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.953322 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:41.953631 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:41.953369 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:41.953631 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:41.953438 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:43.014856 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:43.014809 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"7419d42fafb37e8b2d87d41a6e268c81","Type":"ContainerStarted","Data":"4374886a35c8323924d218089199e7dd03e588a58b73fe136de505d0fa2c596b"} Oct 30 22:00:43.029210 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:43.029096 2583 generic.go:356] "Generic (PLEG): container finished" podID="56d00c89f3615755885c52b7effe8bce" containerID="7db276559323dec53ac72505a6e0b35e52c9461b2a27e548f3227fa3c2f56245" exitCode=0 Oct 30 22:00:43.029210 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:43.029157 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"56d00c89f3615755885c52b7effe8bce","Type":"ContainerDied","Data":"7db276559323dec53ac72505a6e0b35e52c9461b2a27e548f3227fa3c2f56245"} Oct 30 22:00:43.259812 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:43.259740 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" podStartSLOduration=5.259716297 podStartE2EDuration="5.259716297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:43.134607679 +0000 UTC m=+6.722682477" watchObservedRunningTime="2026-10-30 22:00:43.259716297 +0000 UTC m=+6.847791098" Oct 30 22:00:43.952894 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:43.952857 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:43.953086 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:43.952993 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:43.953086 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:43.953073 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:43.953223 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:43.953169 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:43.953300 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:43.953258 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:43.953402 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:43.953381 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:44.038874 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:44.038780 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"56d00c89f3615755885c52b7effe8bce","Type":"ContainerStarted","Data":"0fbb2c1a4d375e6afe4b60873476df8ba0a6d3cf5a2eae6aaa3eea1820012837"} Oct 30 22:00:44.044752 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:44.044722 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="479e3ffb4412d2c5114e9280ddcb7fc0c7fa9e61ef8d5aa82beb180dfe7bf76f" exitCode=0 Oct 30 22:00:44.045345 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:44.045309 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trblz" event={"ID":"1b8fc6dd-36fe-4714-bcda-1863280a8324","Type":"ContainerDied","Data":"479e3ffb4412d2c5114e9280ddcb7fc0c7fa9e61ef8d5aa82beb180dfe7bf76f"} Oct 30 22:00:44.296355 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:44.296231 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" podStartSLOduration=6.296208647 podStartE2EDuration="6.296208647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:44.165061505 +0000 UTC m=+7.753136303" watchObservedRunningTime="2026-10-30 22:00:44.296208647 +0000 UTC m=+7.884283444" Oct 30 22:00:45.064341 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.063925 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4k4pr" event={"ID":"20b0ee9f-f99d-472a-a1d8-df44248d9aec","Type":"ContainerStarted","Data":"2617f6fef554706fd6b09ef585cc19407310cdabd64e755f104fbe59af377f6f"} Oct 30 22:00:45.190559 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.190490 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4k4pr" podStartSLOduration=2.081899818 podStartE2EDuration="7.190469416s" totalImagesPullingTime="5.108569598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:45.189675527 +0000 UTC m=+8.777750323" watchObservedRunningTime="2026-10-30 22:00:45.190469416 +0000 UTC m=+8.778544213" Oct 30 22:00:45.732922 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.732880 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:45.801001 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.800965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:45.801199 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.801043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:45.801199 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.801188 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:45.801358 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.801251 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs podName:cfbc34e7-48ae-456a-ad03-408b46e20053 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:53.80123193 +0000 UTC m=+17.389306707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs") pod "network-metrics-daemon-5qc57" (UID: "cfbc34e7-48ae-456a-ad03-408b46e20053") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:45.801358 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.801342 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:45.801470 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.801382 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret podName:2675c1b0-71d1-40b8-b78c-5ef4a59389d7 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:53.801368715 +0000 UTC m=+17.389443505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret") pod "global-pull-secret-syncer-h8pdj" (UID: "2675c1b0-71d1-40b8-b78c-5ef4a59389d7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:45.901488 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.901442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:45.901674 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.901610 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:45.901674 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.901629 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:45.901674 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.901642 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gzgj8 for pod openshift-network-diagnostics/network-check-target-6mdvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:45.901842 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.901704 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8 podName:588b62a8-9745-4e5a-97dd-f8c451197018 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:53.901683774 +0000 UTC m=+17.489758549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzgj8" (UniqueName: "kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8") pod "network-check-target-6mdvj" (UID: "588b62a8-9745-4e5a-97dd-f8c451197018") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:45.953450 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.953412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:45.953635 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.953554 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:45.953712 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.953636 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:45.953772 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.953718 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:45.953829 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:45.953793 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:45.953912 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:45.953888 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:46.070225 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:46.070145 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-47nqg" event={"ID":"b3b54602-8b76-49b7-b295-42969da3c4fd","Type":"ContainerStarted","Data":"f4270858314641443ac536ea2fc0f4c86f1d39164ebac157ef1ca842c5ab5d5f"} Oct 30 22:00:46.194111 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:46.193417 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-47nqg" podStartSLOduration=2.36873309 podStartE2EDuration="8.193394935s" totalImagesPullingTime="5.824661845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:46.192738832 +0000 UTC m=+9.780813631" watchObservedRunningTime="2026-10-30 22:00:46.193394935 +0000 UTC m=+9.781469734" Oct 30 22:00:47.077133 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:47.077060 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" event={"ID":"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28","Type":"ContainerStarted","Data":"3d263afb095048672f96860d2f274a25cff902eb9ec754b569f191763b21484e"} Oct 30 22:00:47.953236 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:47.953152 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:47.953440 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:47.953313 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:47.953440 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:47.953341 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:47.953570 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:47.953470 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:47.953570 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:47.953537 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:47.953671 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:47.953625 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:48.781271 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:48.781233 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 22:00:48.872517 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:48.871688 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:00:48.78125217Z","UUID":"a9801922-0df6-43de-875d-88f6ef47d79f","Handler":null,"Name":"","Endpoint":""} Oct 30 22:00:48.873482 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:48.873456 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 22:00:48.873597 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:48.873489 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 22:00:49.084223 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.084181 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" event={"ID":"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28","Type":"ContainerStarted","Data":"91418d5e2d0c37c50f26b0793be0c21a18de8fda8524bbbadcc3e0ede04ea74d"} Oct 30 22:00:49.086817 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.086788 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rg6kq" event={"ID":"c986bec8-11a1-4c93-86fa-40b486199598","Type":"ContainerStarted","Data":"0bf2a245c524f4e7b02d3a21c5c6e30f5ee39972378a474a92da7407742362b8"} Oct 30 22:00:49.088790 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.088763 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xc24q" event={"ID":"8f434211-8bee-4801-9c8e-39f0bb0e71d6","Type":"ContainerStarted","Data":"19e7fc0bf3da075bbd0c221c123cd02279d0e75e11f00349092ecbaca5e9236a"} Oct 30 22:00:49.206408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.206352 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rg6kq" podStartSLOduration=1.948770703 podStartE2EDuration="11.206331643s" totalImagesPullingTime="9.25756094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:49.205504888 +0000 UTC m=+12.793579684" watchObservedRunningTime="2026-10-30 22:00:49.206331643 +0000 UTC m=+12.794406442" Oct 30 22:00:49.336568 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.336510 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xc24q" podStartSLOduration=2.676552874 podStartE2EDuration="11.336493208s" totalImagesPullingTime="8.659940334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:49.335687806 +0000 UTC m=+12.923762602" watchObservedRunningTime="2026-10-30 22:00:49.336493208 +0000 UTC m=+12.924568015" Oct 30 22:00:49.906262 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.906224 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:49.907034 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.907013 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:49.952992 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.952959 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:49.952992 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.953004 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:49.953261 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:49.953122 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:49.953261 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:49.953178 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:49.953261 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:49.953202 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:49.953424 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:49.953299 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:50.092908 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:50.092832 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="ea2a3b696913e2bae88aee6ed511f3ed09b23f95c9d170682ad46aca953e4ac3" exitCode=0 Oct 30 22:00:50.092908 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:50.092870 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trblz" event={"ID":"1b8fc6dd-36fe-4714-bcda-1863280a8324","Type":"ContainerDied","Data":"ea2a3b696913e2bae88aee6ed511f3ed09b23f95c9d170682ad46aca953e4ac3"} Oct 30 22:00:50.093898 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:50.093725 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:00:51.097339 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.097244 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" event={"ID":"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28","Type":"ContainerStarted","Data":"b2bed2fd39e72f2b294ef5acc7afe4fa2817b8ccaf1950cafb9269d6fbffcfc2"} Oct 30 22:00:51.099045 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.099016 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c44r7" event={"ID":"8420c80f-804e-468e-8df2-fb1ec145d318","Type":"ContainerStarted","Data":"4651db0d2e5121783f3d155585be5393cdeba87fd3bf7efe28ebf255f0aab507"} Oct 30 22:00:51.100625 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.100599 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qtcg" event={"ID":"94ca8590-27e5-497d-9bb8-cb5041ceb267","Type":"ContainerStarted","Data":"bcc6596300ddec4e9000bd361c471c8fbd21b3aad5dc258159f5fdd39577d3f1"} Oct 30 22:00:51.217065 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.216348 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" podStartSLOduration=2.453264091 podStartE2EDuration="13.21632696s" totalImagesPullingTime="10.763062869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:51.215657249 +0000 UTC m=+14.803732043" watchObservedRunningTime="2026-10-30 22:00:51.21632696 +0000 UTC m=+14.804401754" Oct 30 22:00:51.336547 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.336485 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2qtcg" podStartSLOduration=3.91044999 podStartE2EDuration="13.336465728s" totalImagesPullingTime="9.426015738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:51.336398405 +0000 UTC m=+14.924473199" watchObservedRunningTime="2026-10-30 22:00:51.336465728 +0000 UTC m=+14.924540523" Oct 30 22:00:51.454171 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.454119 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c44r7" podStartSLOduration=2.149662291 podStartE2EDuration="13.454103074s" totalImagesPullingTime="11.304440783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:51.453762369 +0000 UTC m=+15.041837161" watchObservedRunningTime="2026-10-30 22:00:51.454103074 +0000 UTC m=+15.042177867" Oct 30 22:00:51.952712 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.952672 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:51.952889 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.952758 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:51.952889 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:51.952789 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:51.952997 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:51.952913 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:51.953061 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:51.953035 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:51.953174 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:51.953125 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:52.105287 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:52.105249 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="c1bb2944688e30dee0178c7ca7f02d04b70dd01a04f808ec3e9ed528dd63344d" exitCode=0 Oct 30 22:00:52.105836 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:52.105353 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trblz" event={"ID":"1b8fc6dd-36fe-4714-bcda-1863280a8324","Type":"ContainerDied","Data":"c1bb2944688e30dee0178c7ca7f02d04b70dd01a04f808ec3e9ed528dd63344d"} Oct 30 22:00:53.860114 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:53.860065 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:53.860969 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.860251 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:53.860969 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:53.860253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:53.860969 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.860330 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs podName:cfbc34e7-48ae-456a-ad03-408b46e20053 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:09.860309422 +0000 UTC m=+33.448384196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs") pod "network-metrics-daemon-5qc57" (UID: "cfbc34e7-48ae-456a-ad03-408b46e20053") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:00:53.860969 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.860334 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:53.860969 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.860383 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret podName:2675c1b0-71d1-40b8-b78c-5ef4a59389d7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:09.86036892 +0000 UTC m=+33.448443691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret") pod "global-pull-secret-syncer-h8pdj" (UID: "2675c1b0-71d1-40b8-b78c-5ef4a59389d7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:00:53.953178 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:53.953137 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:53.953375 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:53.953226 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:53.953375 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:53.953326 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:53.953375 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.953345 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:53.953500 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.953426 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:53.953536 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.953501 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:53.961288 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:53.961252 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:53.961459 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.961441 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 22:00:53.961505 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.961469 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 22:00:53.961505 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.961483 2583 projected.go:196] Error preparing data for projected volume kube-api-access-gzgj8 for pod openshift-network-diagnostics/network-check-target-6mdvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:53.961585 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:53.961539 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8 podName:588b62a8-9745-4e5a-97dd-f8c451197018 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:09.961520895 +0000 UTC m=+33.549595671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gzgj8" (UniqueName: "kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8") pod "network-check-target-6mdvj" (UID: "588b62a8-9745-4e5a-97dd-f8c451197018") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 22:00:54.112408 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:54.112320 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="ad94505a67e20355a15dcec7434754fb937ba9e23343b15b8624c86e42fdd9e7" exitCode=0 Oct 30 22:00:54.112552 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:54.112405 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trblz" event={"ID":"1b8fc6dd-36fe-4714-bcda-1863280a8324","Type":"ContainerDied","Data":"ad94505a67e20355a15dcec7434754fb937ba9e23343b15b8624c86e42fdd9e7"} Oct 30 22:00:55.116702 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.116523 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:00:55.117117 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.117020 2583 generic.go:356] "Generic (PLEG): container finished" podID="1a72d78e-c521-4dba-b551-5970a7505245" containerID="d10cb107b004efd16c861b75966a655db82f854206f6d5f81b8c2e26d4177553" exitCode=1 Oct 30 22:00:55.117117 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.117051 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"3a3f11d31f2c5df83dae91e75f42a8e005b2ae76ec2953534dd34252fe1a7b91"} Oct 30 22:00:55.117191 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.117135 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"ec2fe0581b31ea20a096e5d79bb3291cc8263cf1e784efd5b2f7a381a04651b6"} Oct 30 22:00:55.117191 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.117175 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerDied","Data":"d10cb107b004efd16c861b75966a655db82f854206f6d5f81b8c2e26d4177553"} Oct 30 22:00:55.117275 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.117220 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"4ff70eeed82bafd43e8bf82330a70af815d9db92870d4a2b85e5a86e0aa2b332"} Oct 30 22:00:55.953090 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.952898 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:55.953245 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.952912 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:55.953245 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:55.953178 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:55.953245 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:55.952942 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:55.953377 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:55.953274 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:55.953419 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:55.953366 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:56.122994 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:56.122965 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:00:56.123459 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:56.123318 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"ae55867bea7538a94534cd5b238212b19e7464f641ade711ba5307fb53dbee6d"} Oct 30 22:00:56.123459 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:56.123385 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"61d6c0296ad762040cd34bc41671e56af5fc8aed1f1b86c2a89a672337944a65"} Oct 30 22:00:57.128337 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:57.128260 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="a964019c4d554a73913785604c10a17851d12dbbe1bb15abb107a99303099ee3" exitCode=0 Oct 30 22:00:57.128337 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:57.128314 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trblz" event={"ID":"1b8fc6dd-36fe-4714-bcda-1863280a8324","Type":"ContainerDied","Data":"a964019c4d554a73913785604c10a17851d12dbbe1bb15abb107a99303099ee3"} Oct 30 22:00:57.953724 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:57.953685 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:57.953905 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:57.953767 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:57.953905 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:57.953854 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:57.953905 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:57.953869 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:57.954018 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:57.953935 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:57.954018 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:57.953999 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:00:58.132716 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:58.132680 2583 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="7d57a31e142d1602e5b2ce3936519b1a128beccd91cfc6a901ee970bb4f32c51" exitCode=0 Oct 30 22:00:58.133199 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:58.132760 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trblz" event={"ID":"1b8fc6dd-36fe-4714-bcda-1863280a8324","Type":"ContainerDied","Data":"7d57a31e142d1602e5b2ce3936519b1a128beccd91cfc6a901ee970bb4f32c51"} Oct 30 22:00:58.135929 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:58.135915 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:00:58.136255 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:58.136224 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"d7087cbbbd12ebfb4e81d1d22cb639fd5ea77601752eac116035bc1117bb9886"} Oct 30 22:00:59.141818 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:59.141778 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trblz" event={"ID":"1b8fc6dd-36fe-4714-bcda-1863280a8324","Type":"ContainerStarted","Data":"393714cc7512aad384447376e78010e24eb606379d2e2b8a53fc4ed69eec532f"} Oct 30 22:00:59.356019 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:59.355969 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-trblz" podStartSLOduration=7.565415451 podStartE2EDuration="21.355953767s" totalImagesPullingTime="13.790538316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:59.355903784 +0000 UTC m=+22.943978579" watchObservedRunningTime="2026-10-30 22:00:59.355953767 +0000 UTC m=+22.944028552" Oct 30 22:00:59.954412 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:59.954231 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:00:59.954628 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:59.954292 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:00:59.954628 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:59.954498 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:00:59.954628 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:00:59.954301 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:00:59.954628 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:59.954553 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:00:59.954825 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:00:59.954658 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:01:00.147401 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:00.147370 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:01:00.148080 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:00.147805 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"fc3023f31cf8c6a728beb2406c2dc8d9fa508ce2fdd3c0fbe48724f86445b1ef"} Oct 30 22:01:00.148476 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:00.148452 2583 scope.go:122] "RemoveContainer" containerID="d10cb107b004efd16c861b75966a655db82f854206f6d5f81b8c2e26d4177553" Oct 30 22:01:01.153818 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.153793 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:01:01.154314 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.154124 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" event={"ID":"1a72d78e-c521-4dba-b551-5970a7505245","Type":"ContainerStarted","Data":"352933146be75d56e14b570033be7de7a5a1333273fa23cf35d495535bd51243"} Oct 30 22:01:01.154456 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.154431 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:01.154532 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.154496 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:01.154606 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.154592 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:01.168817 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.168795 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:01.169829 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.169808 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:01.421962 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.421862 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" podStartSLOduration=8.151278106 podStartE2EDuration="23.421846503s" totalImagesPullingTime="15.270568397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:01.421672354 +0000 UTC m=+25.009747146" watchObservedRunningTime="2026-10-30 22:01:01.421846503 +0000 UTC m=+25.009921296" Oct 30 22:01:01.952732 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.952690 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:01.952925 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.952801 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:01.952925 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:01.952912 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:01:01.953001 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:01.952809 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:01:01.953001 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:01.952808 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:01.953001 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:01.952988 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:01:02.263150 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:02.263109 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6mdvj"] Oct 30 22:01:02.263666 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:02.263265 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:02.263666 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:02.263364 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:01:02.266294 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:02.266266 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h8pdj"] Oct 30 22:01:02.266418 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:02.266361 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:02.266496 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:02.266456 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:01:02.266795 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:02.266775 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qc57"] Oct 30 22:01:02.266887 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:02.266874 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:02.266996 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:02.266977 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:01:03.953603 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:03.953572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:03.953603 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:03.953593 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:03.954010 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:03.953593 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:03.954010 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:03.953678 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:01:03.954010 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:03.953773 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:01:03.954010 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:03.953858 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:01:05.953671 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:05.953641 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:05.953671 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:05.953670 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:05.954067 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:05.953736 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:05.954067 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:05.953752 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:01:05.954067 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:05.953816 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:01:05.954067 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:05.953891 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:01:07.953440 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:07.953256 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:07.953835 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:07.953256 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:07.953835 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:07.953524 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qc57" podUID="cfbc34e7-48ae-456a-ad03-408b46e20053" Oct 30 22:01:07.953835 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:07.953264 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:07.953835 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:07.953589 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h8pdj" podUID="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" Oct 30 22:01:07.953835 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:07.953664 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mdvj" podUID="588b62a8-9745-4e5a-97dd-f8c451197018" Oct 30 22:01:08.172912 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.172885 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeReady" Oct 30 22:01:08.173046 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.173004 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 22:01:08.296522 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.296484 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l6v6f"] Oct 30 22:01:08.298480 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.298464 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l4pxs"] Oct 30 22:01:08.298599 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.298582 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.300141 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.300123 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:08.321307 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.321282 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5w4t9"] Oct 30 22:01:08.323520 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.323503 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.351187 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.351165 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 22:01:08.351557 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.351542 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 22:01:08.351651 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.351581 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pd25c\"" Oct 30 22:01:08.356987 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.356971 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 22:01:08.362948 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.361950 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l6v6f"] Oct 30 22:01:08.362948 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.362069 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l4pxs"] Oct 30 22:01:08.362948 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.362595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/089436e9-8607-4901-9b23-8e0512aacc3c-config-volume\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.362948 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.362692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.362948 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.362792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d531260-0f02-4b21-af0a-ec862fca540d-cert\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:08.362948 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.362842 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsdpq\" (UniqueName: \"kubernetes.io/projected/8d531260-0f02-4b21-af0a-ec862fca540d-kube-api-access-wsdpq\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:08.362948 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.362880 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089436e9-8607-4901-9b23-8e0512aacc3c-tmp-dir\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.362948 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.362904 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvlp\" (UniqueName: \"kubernetes.io/projected/089436e9-8607-4901-9b23-8e0512aacc3c-kube-api-access-lgvlp\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.423138 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.423072 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5w4t9"] Oct 30 22:01:08.452748 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.452718 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 22:01:08.452931 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.452900 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 22:01:08.452931 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.452914 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sqg6l\"" Oct 30 22:01:08.453117 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.452972 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 22:01:08.453117 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.452987 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g5t42\"" Oct 30 22:01:08.453117 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.453001 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 22:01:08.453117 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.453002 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 22:01:08.453251 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.453136 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 22:01:08.464211 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464189 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.464281 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d531260-0f02-4b21-af0a-ec862fca540d-cert\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:08.464281 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsdpq\" (UniqueName: \"kubernetes.io/projected/8d531260-0f02-4b21-af0a-ec862fca540d-kube-api-access-wsdpq\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:08.464281 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464277 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089436e9-8607-4901-9b23-8e0512aacc3c-tmp-dir\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.464421 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvlp\" (UniqueName: \"kubernetes.io/projected/089436e9-8607-4901-9b23-8e0512aacc3c-kube-api-access-lgvlp\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.464421 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:08.464335 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 22:01:08.464523 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:08.464449 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls podName:089436e9-8607-4901-9b23-8e0512aacc3c nodeName:}" failed. No retries permitted until 2026-10-30 22:01:08.964426673 +0000 UTC m=+32.552501450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls") pod "dns-default-l6v6f" (UID: "089436e9-8607-4901-9b23-8e0512aacc3c") : secret "dns-default-metrics-tls" not found Oct 30 22:01:08.464625 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.464680 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464631 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba98bdc2-2e27-4499-8182-4abf75b94a23-data-volume\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.464735 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.464735 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464725 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089436e9-8607-4901-9b23-8e0512aacc3c-tmp-dir\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.464796 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppgx\" (UniqueName: \"kubernetes.io/projected/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-api-access-kppgx\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.464830 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464793 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/089436e9-8607-4901-9b23-8e0512aacc3c-config-volume\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.464871 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.464827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba98bdc2-2e27-4499-8182-4abf75b94a23-crio-socket\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.465250 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.465233 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/089436e9-8607-4901-9b23-8e0512aacc3c-config-volume\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.468393 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.468378 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d531260-0f02-4b21-af0a-ec862fca540d-cert\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:08.544677 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.544647 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsdpq\" (UniqueName: \"kubernetes.io/projected/8d531260-0f02-4b21-af0a-ec862fca540d-kube-api-access-wsdpq\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:08.547216 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.547180 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvlp\" (UniqueName: \"kubernetes.io/projected/089436e9-8607-4901-9b23-8e0512aacc3c-kube-api-access-lgvlp\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.566169 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.566136 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba98bdc2-2e27-4499-8182-4abf75b94a23-data-volume\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.566336 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.566183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.566336 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.566217 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppgx\" (UniqueName: \"kubernetes.io/projected/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-api-access-kppgx\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.566336 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:08.566300 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 22:01:08.566510 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.566336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba98bdc2-2e27-4499-8182-4abf75b94a23-crio-socket\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.566510 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.566438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.566510 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.566498 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba98bdc2-2e27-4499-8182-4abf75b94a23-data-volume\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.566685 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:08.566499 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm podName:ba98bdc2-2e27-4499-8182-4abf75b94a23 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:09.066470497 +0000 UTC m=+32.654545276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5w4t9" (UID: "ba98bdc2-2e27-4499-8182-4abf75b94a23") : configmap "kube-rbac-proxy" not found Oct 30 22:01:08.566685 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:08.566556 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 22:01:08.566685 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.566600 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba98bdc2-2e27-4499-8182-4abf75b94a23-crio-socket\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.566685 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:08.566618 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls podName:ba98bdc2-2e27-4499-8182-4abf75b94a23 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:09.066603465 +0000 UTC m=+32.654678255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w4t9" (UID: "ba98bdc2-2e27-4499-8182-4abf75b94a23") : secret "insights-runtime-extractor-tls" not found Oct 30 22:01:08.614493 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.614464 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:08.626662 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.626636 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppgx\" (UniqueName: \"kubernetes.io/projected/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-api-access-kppgx\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:08.969950 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.969872 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:08.970410 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:08.970025 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 22:01:08.970410 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:08.970096 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls podName:089436e9-8607-4901-9b23-8e0512aacc3c nodeName:}" failed. No retries permitted until 2026-10-30 22:01:09.970078494 +0000 UTC m=+33.558153282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls") pod "dns-default-l6v6f" (UID: "089436e9-8607-4901-9b23-8e0512aacc3c") : secret "dns-default-metrics-tls" not found Oct 30 22:01:08.998694 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:08.998666 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l4pxs"] Oct 30 22:01:09.004963 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:09.004938 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d531260_0f02_4b21_af0a_ec862fca540d.slice/crio-ae1048f517c07efa6ea8af000c9ba83fd79df54c9c4b6e01e9006254d50191fb WatchSource:0}: Error finding container ae1048f517c07efa6ea8af000c9ba83fd79df54c9c4b6e01e9006254d50191fb: Status 404 returned error can't find the container with id ae1048f517c07efa6ea8af000c9ba83fd79df54c9c4b6e01e9006254d50191fb Oct 30 22:01:09.070814 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.070774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:09.070990 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.070850 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:09.070990 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:09.070949 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 22:01:09.071070 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:09.070998 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls podName:ba98bdc2-2e27-4499-8182-4abf75b94a23 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:10.070986172 +0000 UTC m=+33.659060943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w4t9" (UID: "ba98bdc2-2e27-4499-8182-4abf75b94a23") : secret "insights-runtime-extractor-tls" not found Oct 30 22:01:09.071326 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.071304 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:09.173092 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.173055 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l4pxs" event={"ID":"8d531260-0f02-4b21-af0a-ec862fca540d","Type":"ContainerStarted","Data":"ae1048f517c07efa6ea8af000c9ba83fd79df54c9c4b6e01e9006254d50191fb"} Oct 30 22:01:09.715221 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.715183 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4k4pr_20b0ee9f-f99d-472a-a1d8-df44248d9aec/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:09.878041 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.878006 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:09.878173 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.878062 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:09.878173 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:09.878158 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 22:01:09.878173 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:09.878160 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:01:09.878305 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:09.878282 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret podName:2675c1b0-71d1-40b8-b78c-5ef4a59389d7 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:41.878196023 +0000 UTC m=+65.466270807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret") pod "global-pull-secret-syncer-h8pdj" (UID: "2675c1b0-71d1-40b8-b78c-5ef4a59389d7") : object "kube-system"/"original-pull-secret" not registered Oct 30 22:01:09.878353 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:09.878312 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs podName:cfbc34e7-48ae-456a-ad03-408b46e20053 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:41.878301394 +0000 UTC m=+65.466376169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs") pod "network-metrics-daemon-5qc57" (UID: "cfbc34e7-48ae-456a-ad03-408b46e20053") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 22:01:09.953726 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.953694 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:09.953826 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.953762 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:09.953826 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.953789 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:09.978738 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.978715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:09.979094 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:09.978777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:09.979094 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:09.978882 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 22:01:09.979094 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:09.978952 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls podName:089436e9-8607-4901-9b23-8e0512aacc3c nodeName:}" failed. No retries permitted until 2026-10-30 22:01:11.978932683 +0000 UTC m=+35.567007468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls") pod "dns-default-l6v6f" (UID: "089436e9-8607-4901-9b23-8e0512aacc3c") : secret "dns-default-metrics-tls" not found Oct 30 22:01:10.004723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.004695 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 22:01:10.004723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.004695 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rstdk\"" Oct 30 22:01:10.004869 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.004708 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 22:01:10.024339 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.024306 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4k4pr_20b0ee9f-f99d-472a-a1d8-df44248d9aec/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:10.031736 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.031719 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 22:01:10.042345 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.042314 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:10.052297 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.052268 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-skl9p\"" Oct 30 22:01:10.052297 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.052277 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6j7ll\"" Oct 30 22:01:10.052439 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.052355 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 22:01:10.079739 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.079701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:10.079922 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:10.079830 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 22:01:10.079922 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:10.079905 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls podName:ba98bdc2-2e27-4499-8182-4abf75b94a23 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:12.079882708 +0000 UTC m=+35.667957478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5w4t9" (UID: "ba98bdc2-2e27-4499-8182-4abf75b94a23") : secret "insights-runtime-extractor-tls" not found Oct 30 22:01:10.271250 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.271157 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:10.637611 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:10.637537 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6mdvj"] Oct 30 22:01:10.644653 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:10.644627 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588b62a8_9745_4e5a_97dd_f8c451197018.slice/crio-02c601c74d0744c0328576bd161bc5534ed6d4c6f9dca96aa9cf17f70399debf WatchSource:0}: Error finding container 02c601c74d0744c0328576bd161bc5534ed6d4c6f9dca96aa9cf17f70399debf: Status 404 returned error can't find the container with id 02c601c74d0744c0328576bd161bc5534ed6d4c6f9dca96aa9cf17f70399debf Oct 30 22:01:11.183216 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:11.183168 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6mdvj" event={"ID":"588b62a8-9745-4e5a-97dd-f8c451197018","Type":"ContainerStarted","Data":"02c601c74d0744c0328576bd161bc5534ed6d4c6f9dca96aa9cf17f70399debf"} Oct 30 22:01:11.723746 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:11.723713 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rg6kq_c986bec8-11a1-4c93-86fa-40b486199598/dns-node-resolver/0.log" Oct 30 22:01:11.995305 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:11.995265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:11.995493 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:11.995445 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 22:01:11.995557 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:11.995532 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls podName:089436e9-8607-4901-9b23-8e0512aacc3c nodeName:}" failed. No retries permitted until 2026-10-30 22:01:15.995510694 +0000 UTC m=+39.583585485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls") pod "dns-default-l6v6f" (UID: "089436e9-8607-4901-9b23-8e0512aacc3c") : secret "dns-default-metrics-tls" not found Oct 30 22:01:12.096705 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:12.096619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:12.099503 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:12.099469 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:12.235074 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:12.235041 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:12.322879 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:12.322853 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-47nqg_b3b54602-8b76-49b7-b295-42969da3c4fd/node-ca/0.log" Oct 30 22:01:12.614990 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:12.614950 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5w4t9"] Oct 30 22:01:12.649068 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:12.649037 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba98bdc2_2e27_4499_8182_4abf75b94a23.slice/crio-8ca9772597ceae78fddadb74ae19168082d214b4a2ae820a325ccddf54578e52 WatchSource:0}: Error finding container 8ca9772597ceae78fddadb74ae19168082d214b4a2ae820a325ccddf54578e52: Status 404 returned error can't find the container with id 8ca9772597ceae78fddadb74ae19168082d214b4a2ae820a325ccddf54578e52 Oct 30 22:01:13.190574 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:13.190536 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l4pxs" event={"ID":"8d531260-0f02-4b21-af0a-ec862fca540d","Type":"ContainerStarted","Data":"f01d0537816d85475f9b311490c10511f703496b76aad6f43732fd4e1fd3d424"} Oct 30 22:01:13.192219 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:13.192184 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5w4t9" event={"ID":"ba98bdc2-2e27-4499-8182-4abf75b94a23","Type":"ContainerStarted","Data":"559f53db3358503b65b1336d920ee4b954e9b3ef7d437fbcee1e97d485ef81fa"} Oct 30 22:01:13.192344 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:13.192244 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5w4t9" event={"ID":"ba98bdc2-2e27-4499-8182-4abf75b94a23","Type":"ContainerStarted","Data":"8ca9772597ceae78fddadb74ae19168082d214b4a2ae820a325ccddf54578e52"} Oct 30 22:01:13.306484 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:13.306435 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l4pxs" podStartSLOduration=1.883657972 podStartE2EDuration="5.306421598s" totalImagesPullingTime="3.422763626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:13.306310788 +0000 UTC m=+36.894385582" watchObservedRunningTime="2026-10-30 22:01:13.306421598 +0000 UTC m=+36.894496382" Oct 30 22:01:14.197116 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:14.197080 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5w4t9" event={"ID":"ba98bdc2-2e27-4499-8182-4abf75b94a23","Type":"ContainerStarted","Data":"fcb80729e1cd9a45652ffac29674716463d76350011a3e0f4b1d0c51b7cb9795"} Oct 30 22:01:15.201927 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:15.201886 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6mdvj" event={"ID":"588b62a8-9745-4e5a-97dd-f8c451197018","Type":"ContainerStarted","Data":"96aa909fc937a01ddda048f91dc0472a8f1552b940af33e8104a1552e6685950"} Oct 30 22:01:15.202415 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:15.202016 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:15.312453 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:15.312397 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6mdvj" podStartSLOduration=33.693365831 podStartE2EDuration="37.312381904s" totalImagesPullingTime="3.619016073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:15.312086298 +0000 UTC m=+38.900161090" watchObservedRunningTime="2026-10-30 22:01:15.312381904 +0000 UTC m=+38.900456724" Oct 30 22:01:16.028536 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:16.028495 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:16.028675 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:16.028640 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 22:01:16.028715 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:16.028703 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls podName:089436e9-8607-4901-9b23-8e0512aacc3c nodeName:}" failed. No retries permitted until 2026-10-30 22:01:24.028688509 +0000 UTC m=+47.616763299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls") pod "dns-default-l6v6f" (UID: "089436e9-8607-4901-9b23-8e0512aacc3c") : secret "dns-default-metrics-tls" not found Oct 30 22:01:16.206915 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:16.206830 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5w4t9" event={"ID":"ba98bdc2-2e27-4499-8182-4abf75b94a23","Type":"ContainerStarted","Data":"21d2d503067c01987c781c731b081a4cfec9cc95d4ae9c1bf7b3c0066ab5bf0d"} Oct 30 22:01:16.321724 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:16.321670 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5w4t9" podStartSLOduration=5.193587109 podStartE2EDuration="8.321655911s" totalImagesPullingTime="3.128068802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:01:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:16.320621396 +0000 UTC m=+39.908696209" watchObservedRunningTime="2026-10-30 22:01:16.321655911 +0000 UTC m=+39.909730703" Oct 30 22:01:24.086940 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:24.086902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:24.089191 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:24.089165 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:24.211606 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:24.211571 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:24.657423 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:24.657345 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l6v6f"] Oct 30 22:01:24.664309 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:24.664277 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089436e9_8607_4901_9b23_8e0512aacc3c.slice/crio-ed1d70f775129e8c5081b6c9c35aeaca64eff12ade600d9b2540fc739b58901e WatchSource:0}: Error finding container ed1d70f775129e8c5081b6c9c35aeaca64eff12ade600d9b2540fc739b58901e: Status 404 returned error can't find the container with id ed1d70f775129e8c5081b6c9c35aeaca64eff12ade600d9b2540fc739b58901e Oct 30 22:01:25.230322 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:25.230290 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l6v6f" event={"ID":"089436e9-8607-4901-9b23-8e0512aacc3c","Type":"ContainerStarted","Data":"ed1d70f775129e8c5081b6c9c35aeaca64eff12ade600d9b2540fc739b58901e"} Oct 30 22:01:26.763292 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.763256 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-949b9f975-65pwb"] Oct 30 22:01:26.777139 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.777111 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.826171 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.826135 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-949b9f975-65pwb"] Oct 30 22:01:26.907402 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.907365 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-trusted-ca\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.907402 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.907402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-installation-pull-secrets\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.907605 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.907439 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-ca-trust-extracted\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.907605 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.907457 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l64c\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-kube-api-access-2l64c\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.907605 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.907476 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-image-registry-private-configuration\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.907605 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.907492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-bound-sa-token\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.907605 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.907526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-certificates\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.907605 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.907543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-tls\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:26.938617 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.938585 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 22:01:26.967485 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.967453 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 22:01:26.967654 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.967540 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 22:01:26.967654 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.967578 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 22:01:26.967728 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:26.967708 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hpnbl\"" Oct 30 22:01:27.008299 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-trusted-ca\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.008299 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-installation-pull-secrets\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.008531 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-ca-trust-extracted\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.008531 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008359 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l64c\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-kube-api-access-2l64c\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.008531 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008388 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-image-registry-private-configuration\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.008531 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008413 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-bound-sa-token\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.008531 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-certificates\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.008754 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-tls\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.008809 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.008776 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-ca-trust-extracted\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.009508 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.009482 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-certificates\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.009699 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.009676 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-trusted-ca\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.011054 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.011020 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-installation-pull-secrets\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.011175 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.011134 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-tls\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.011311 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.011178 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-image-registry-private-configuration\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.070551 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.070489 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l64c\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-kube-api-access-2l64c\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.070674 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.070569 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-bound-sa-token\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.097888 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.097864 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:27.530081 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:27.530043 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-949b9f975-65pwb"] Oct 30 22:01:27.537523 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:27.537493 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef08e87_e62d_4ff9_9b6c_e90c7bf48b70.slice/crio-a8097e11515b58ec89c2a45e643c054bdd57dad923ba8908b96adcfb5fdf41a1 WatchSource:0}: Error finding container a8097e11515b58ec89c2a45e643c054bdd57dad923ba8908b96adcfb5fdf41a1: Status 404 returned error can't find the container with id a8097e11515b58ec89c2a45e643c054bdd57dad923ba8908b96adcfb5fdf41a1 Oct 30 22:01:28.239312 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:28.239260 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-949b9f975-65pwb" event={"ID":"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70","Type":"ContainerStarted","Data":"0b3a221ae2a03018856b7e8a0cf280e3d473537532736a4b162d158773b9128e"} Oct 30 22:01:28.239713 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:28.239367 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:28.239713 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:28.239392 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-949b9f975-65pwb" event={"ID":"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70","Type":"ContainerStarted","Data":"a8097e11515b58ec89c2a45e643c054bdd57dad923ba8908b96adcfb5fdf41a1"} Oct 30 22:01:28.240893 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:28.240871 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l6v6f" event={"ID":"089436e9-8607-4901-9b23-8e0512aacc3c","Type":"ContainerStarted","Data":"e2a9639341046545302af90578e0c3b8c4a02a3af17dd3b0b5b1125fb73803c3"} Oct 30 22:01:28.241003 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:28.240910 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l6v6f" event={"ID":"089436e9-8607-4901-9b23-8e0512aacc3c","Type":"ContainerStarted","Data":"7379b052049aefff0487c74eeb06b393c8442a5242c171d2a02c6c79181ccc06"} Oct 30 22:01:28.241041 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:28.241000 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:28.356499 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:28.356452 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-949b9f975-65pwb" podStartSLOduration=2.356438553 podStartE2EDuration="2.356438553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:28.355895939 +0000 UTC m=+51.943970730" watchObservedRunningTime="2026-10-30 22:01:28.356438553 +0000 UTC m=+51.944513345" Oct 30 22:01:28.469985 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:28.469926 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-l6v6f" podStartSLOduration=17.230609943 podStartE2EDuration="20.46991138s" totalImagesPullingTime="3.239301437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:28.468974842 +0000 UTC m=+52.057049653" watchObservedRunningTime="2026-10-30 22:01:28.46991138 +0000 UTC m=+52.057986172" Oct 30 22:01:33.169520 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:33.169493 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:38.246577 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:38.246543 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:41.912803 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:41.912754 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:41.913276 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:41.912822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:41.965511 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:41.965477 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 22:01:41.975889 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:41.975864 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:42.012899 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.012863 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 22:01:42.015270 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.015246 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:42.116157 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.116130 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-skl9p\"" Oct 30 22:01:42.118253 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.118234 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:42.125709 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.125687 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6j7ll\"" Oct 30 22:01:42.135827 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.135799 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:42.261457 ip-10-0-141-194 kubenswrapper[2583]: E1030 22:01:42.261422 2583 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:01:42.487677 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.487628 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qc57"] Oct 30 22:01:42.532107 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:42.532082 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbc34e7_48ae_456a_ad03_408b46e20053.slice/crio-7ae20dfd94b056b365e689e2cb06a6a7b079ae3d37f3f2cfb3949d55fa19a987 WatchSource:0}: Error finding container 7ae20dfd94b056b365e689e2cb06a6a7b079ae3d37f3f2cfb3949d55fa19a987: Status 404 returned error can't find the container with id 7ae20dfd94b056b365e689e2cb06a6a7b079ae3d37f3f2cfb3949d55fa19a987 Oct 30 22:01:42.542879 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.542853 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h8pdj"] Oct 30 22:01:42.549559 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:42.549532 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2675c1b0_71d1_40b8_b78c_5ef4a59389d7.slice/crio-8cd6cd6d89bf397e6c645743679bb13189d8ee69d1ca5fdc1536d52a56b37d93 WatchSource:0}: Error finding container 8cd6cd6d89bf397e6c645743679bb13189d8ee69d1ca5fdc1536d52a56b37d93: Status 404 returned error can't find the container with id 8cd6cd6d89bf397e6c645743679bb13189d8ee69d1ca5fdc1536d52a56b37d93 Oct 30 22:01:42.648633 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.648599 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h7dlr"] Oct 30 22:01:42.652106 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.652075 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.702119 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.702083 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 22:01:42.702119 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.702083 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qp8bv\"" Oct 30 22:01:42.702355 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.702190 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 22:01:42.718112 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718078 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-root\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.718234 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.718234 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718157 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.718309 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718247 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-tls\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.718309 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhtx\" (UniqueName: \"kubernetes.io/projected/3f522003-5ca0-45d8-b32e-b52db6916aca-kube-api-access-5bhtx\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.718309 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718284 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-sys\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.718309 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718300 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-textfile\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.718489 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718335 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-wtmp\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.718489 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.718430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-metrics-client-ca\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.749629 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.749551 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 22:01:42.749629 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.749551 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 22:01:42.749629 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.749610 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 22:01:42.749629 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.749647 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 22:01:42.818879 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.818838 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-textfile\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.818879 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.818877 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-wtmp\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819085 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.818892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-metrics-client-ca\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819085 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.818932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-root\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819085 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.818977 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819085 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819085 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819033 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-tls\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819085 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819052 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-wtmp\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819085 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819058 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-root\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819085 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhtx\" (UniqueName: \"kubernetes.io/projected/3f522003-5ca0-45d8-b32e-b52db6916aca-kube-api-access-5bhtx\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819510 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819249 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-sys\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819510 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819264 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-textfile\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819510 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819196 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-sys\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819641 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819619 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-metrics-client-ca\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.819694 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.819619 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.821336 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.821312 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-tls\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.821443 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.821390 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.882741 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.882706 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhtx\" (UniqueName: \"kubernetes.io/projected/3f522003-5ca0-45d8-b32e-b52db6916aca-kube-api-access-5bhtx\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.965237 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:42.965188 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:42.974837 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:42.974813 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f522003_5ca0_45d8_b32e_b52db6916aca.slice/crio-08dad873475d087043c9cf3f64513901ba39638bff4b6e9205dbe8657deb1e15 WatchSource:0}: Error finding container 08dad873475d087043c9cf3f64513901ba39638bff4b6e9205dbe8657deb1e15: Status 404 returned error can't find the container with id 08dad873475d087043c9cf3f64513901ba39638bff4b6e9205dbe8657deb1e15 Oct 30 22:01:43.286766 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:43.286725 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7dlr" event={"ID":"3f522003-5ca0-45d8-b32e-b52db6916aca","Type":"ContainerStarted","Data":"08dad873475d087043c9cf3f64513901ba39638bff4b6e9205dbe8657deb1e15"} Oct 30 22:01:43.288351 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:43.288312 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qc57" event={"ID":"cfbc34e7-48ae-456a-ad03-408b46e20053","Type":"ContainerStarted","Data":"7ae20dfd94b056b365e689e2cb06a6a7b079ae3d37f3f2cfb3949d55fa19a987"} Oct 30 22:01:43.289418 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:43.289395 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h8pdj" event={"ID":"2675c1b0-71d1-40b8-b78c-5ef4a59389d7","Type":"ContainerStarted","Data":"8cd6cd6d89bf397e6c645743679bb13189d8ee69d1ca5fdc1536d52a56b37d93"} Oct 30 22:01:45.045913 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.045877 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:01:45.050332 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.050299 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.106594 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.106542 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 22:01:45.106594 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.106576 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 22:01:45.106816 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.106600 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 22:01:45.133305 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133271 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133495 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133313 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133495 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133342 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133495 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133421 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133652 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133652 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133548 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gprrl\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133652 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133570 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133652 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133591 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133652 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133625 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133880 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133880 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.133880 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133854 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.134000 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.133969 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.154011 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.153983 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 22:01:45.202054 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.202015 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 22:01:45.202054 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.202019 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 22:01:45.202385 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.202016 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 22:01:45.202385 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.202087 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 22:01:45.204546 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.204521 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:01:45.206533 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.206505 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 22:01:45.206625 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.206608 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v5kds\"" Oct 30 22:01:45.235048 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235013 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235167 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235053 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235167 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235167 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235167 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235143 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235349 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235349 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235349 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235244 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235349 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235266 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235349 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235294 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235349 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235324 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235349 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235345 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gprrl\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235671 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235366 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.235741 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.235718 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.236376 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.236227 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.237049 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.237019 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.238374 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.238331 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.238467 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.238431 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.239533 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.239514 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.239943 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.239918 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.240166 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.240148 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.240225 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.240190 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.240359 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.240338 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.241339 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.241318 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.241845 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.241822 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.296262 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.296172 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gprrl\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.300702 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.300597 2583 generic.go:356] "Generic (PLEG): container finished" podID="3f522003-5ca0-45d8-b32e-b52db6916aca" containerID="c180196eeec6d25c1315c2abc16b5a4c02c5c0b7df639e2d2cba113e163b31c9" exitCode=0 Oct 30 22:01:45.300702 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.300694 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7dlr" event={"ID":"3f522003-5ca0-45d8-b32e-b52db6916aca","Type":"ContainerDied","Data":"c180196eeec6d25c1315c2abc16b5a4c02c5c0b7df639e2d2cba113e163b31c9"} Oct 30 22:01:45.304784 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.304757 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qc57" event={"ID":"cfbc34e7-48ae-456a-ad03-408b46e20053","Type":"ContainerStarted","Data":"7dc69f1be05fdf051c8153416c89d0f10500866ee02108add7ddd07080ef5957"} Oct 30 22:01:45.304895 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.304808 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qc57" event={"ID":"cfbc34e7-48ae-456a-ad03-408b46e20053","Type":"ContainerStarted","Data":"b47508606887e117b6d872fd236edda6a0367627092d6406a55ef54773aa516b"} Oct 30 22:01:45.371463 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.371427 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:45.375603 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.374418 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-665bfc67b6-srdl5"] Oct 30 22:01:45.377475 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.377455 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.429981 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.429951 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 22:01:45.429981 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.429965 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 22:01:45.430170 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.430047 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 22:01:45.430251 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.430239 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 22:01:45.430313 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.430268 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-122dschqev18o\"" Oct 30 22:01:45.430363 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.430317 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 22:01:45.431009 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.430988 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rw2ld\"" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.436944 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9bbq\" (UniqueName: \"kubernetes.io/projected/10f4cba6-57dc-4397-a602-21a3bad21b82-kube-api-access-d9bbq\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.436986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10f4cba6-57dc-4397-a602-21a3bad21b82-metrics-client-ca\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.437035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.437065 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.437092 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.437130 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.437166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.437241 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f4cba6-57dc-4397-a602-21a3bad21b82-tmp\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.437268 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-grpc-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.439711 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.439409 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-665bfc67b6-srdl5"] Oct 30 22:01:45.538665 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.538631 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f4cba6-57dc-4397-a602-21a3bad21b82-tmp\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.538665 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.538672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-grpc-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.539115 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539087 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f4cba6-57dc-4397-a602-21a3bad21b82-tmp\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.539231 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539147 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9bbq\" (UniqueName: \"kubernetes.io/projected/10f4cba6-57dc-4397-a602-21a3bad21b82-kube-api-access-d9bbq\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.539231 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539185 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10f4cba6-57dc-4397-a602-21a3bad21b82-metrics-client-ca\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.539337 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.539337 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.539337 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539308 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.539488 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539345 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.539488 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539378 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.540255 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.539967 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10f4cba6-57dc-4397-a602-21a3bad21b82-metrics-client-ca\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.541940 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.541901 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.542534 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.542512 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.542659 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.542642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.542857 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.542834 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-grpc-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.543063 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.543042 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.544022 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.544005 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.599884 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.599807 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9bbq\" (UniqueName: \"kubernetes.io/projected/10f4cba6-57dc-4397-a602-21a3bad21b82-kube-api-access-d9bbq\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.645467 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.645410 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5qc57" podStartSLOduration=65.97697391 podStartE2EDuration="1m7.645395844s" totalImagesPullingTime="1.668421934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:45.645045219 +0000 UTC m=+69.233120015" watchObservedRunningTime="2026-10-30 22:01:45.645395844 +0000 UTC m=+69.233470637" Oct 30 22:01:45.714193 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.714155 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:45.814693 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:45.814655 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:01:45.820469 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:45.820437 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd5821b_7635_4bbf_9e07_86fa1579328d.slice/crio-7552fcdfe513c4d4a5e6b2333faf9987c0c4359fa2ca9681c30eaa331d192049 WatchSource:0}: Error finding container 7552fcdfe513c4d4a5e6b2333faf9987c0c4359fa2ca9681c30eaa331d192049: Status 404 returned error can't find the container with id 7552fcdfe513c4d4a5e6b2333faf9987c0c4359fa2ca9681c30eaa331d192049 Oct 30 22:01:46.122433 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:46.122382 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-665bfc67b6-srdl5"] Oct 30 22:01:46.129743 ip-10-0-141-194 kubenswrapper[2583]: W1030 22:01:46.129704 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f4cba6_57dc_4397_a602_21a3bad21b82.slice/crio-8eaa2d97e2fb5b7f114b775d094f69c7bec98c2e3e155f90d1dc8ce7cffc5182 WatchSource:0}: Error finding container 8eaa2d97e2fb5b7f114b775d094f69c7bec98c2e3e155f90d1dc8ce7cffc5182: Status 404 returned error can't find the container with id 8eaa2d97e2fb5b7f114b775d094f69c7bec98c2e3e155f90d1dc8ce7cffc5182 Oct 30 22:01:46.210041 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:46.210011 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:46.310800 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:46.310711 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7dlr" event={"ID":"3f522003-5ca0-45d8-b32e-b52db6916aca","Type":"ContainerStarted","Data":"d1930b4e5fb96f127d6cdd56594d6b76f73e27e61bab7469f6f81321bc8d5aa2"} Oct 30 22:01:46.310800 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:46.310771 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7dlr" event={"ID":"3f522003-5ca0-45d8-b32e-b52db6916aca","Type":"ContainerStarted","Data":"b3d0bd23e648d3cc627caacb7073c7c251d52532a65f009757bb0ca5cbb98641"} Oct 30 22:01:46.312377 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:46.312339 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerStarted","Data":"7552fcdfe513c4d4a5e6b2333faf9987c0c4359fa2ca9681c30eaa331d192049"} Oct 30 22:01:46.313761 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:46.313732 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" event={"ID":"10f4cba6-57dc-4397-a602-21a3bad21b82","Type":"ContainerStarted","Data":"8eaa2d97e2fb5b7f114b775d094f69c7bec98c2e3e155f90d1dc8ce7cffc5182"} Oct 30 22:01:46.436094 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:46.436032 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h7dlr" podStartSLOduration=2.762584228 podStartE2EDuration="4.436014236s" totalImagesPullingTime="1.595528138s" totalInitContainerRuntime="77.90187ms" isStatefulPod=true podCreationTimestamp="2026-10-30 22:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:46.435134807 +0000 UTC m=+70.023209601" watchObservedRunningTime="2026-10-30 22:01:46.436014236 +0000 UTC m=+70.024089029" Oct 30 22:01:47.555091 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.555052 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-cbb75f5c9-k9gj6"] Oct 30 22:01:47.574229 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.574116 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.614723 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.614631 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cbb75f5c9-k9gj6"] Oct 30 22:01:47.629558 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.629522 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 22:01:47.629785 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.629767 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 22:01:47.629897 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.629881 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 22:01:47.629988 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.629974 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 22:01:47.630125 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.630104 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wf9b4\"" Oct 30 22:01:47.630174 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.630127 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1pbn1s61uifeu\"" Oct 30 22:01:47.659748 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.659681 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-client-certs\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.659936 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.659778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3058b5f8-ae0e-4193-b4dd-6d4abace3282-audit-log\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.659936 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.659802 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-tls\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.659936 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.659836 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-client-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.660098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.660007 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.660098 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.660074 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbrr\" (UniqueName: \"kubernetes.io/projected/3058b5f8-ae0e-4193-b4dd-6d4abace3282-kube-api-access-wpbrr\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.660190 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.660110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-metrics-server-audit-profiles\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.761570 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.761524 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbrr\" (UniqueName: \"kubernetes.io/projected/3058b5f8-ae0e-4193-b4dd-6d4abace3282-kube-api-access-wpbrr\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.761753 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.761580 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-metrics-server-audit-profiles\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.761753 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.761615 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-client-certs\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.761753 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.761655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3058b5f8-ae0e-4193-b4dd-6d4abace3282-audit-log\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.761753 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.761683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-tls\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.761753 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.761721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-client-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.761929 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.761777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.762493 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.762120 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3058b5f8-ae0e-4193-b4dd-6d4abace3282-audit-log\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.762602 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.762551 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.762773 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.762750 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-metrics-server-audit-profiles\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.764889 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.764866 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-tls\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.765189 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.765162 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-client-certs\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.766861 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.766838 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-client-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.826259 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.826168 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbrr\" (UniqueName: \"kubernetes.io/projected/3058b5f8-ae0e-4193-b4dd-6d4abace3282-kube-api-access-wpbrr\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:47.900993 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:47.900948 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:48.356471 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.356434 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d"] Oct 30 22:01:48.377322 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.376969 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.432178 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.432152 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 22:01:48.467487 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.467452 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.467663 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.467507 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.467663 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.467598 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ccz\" (UniqueName: \"kubernetes.io/projected/9dce0456-dccb-4e0f-bdbd-978371813dc4-kube-api-access-l4ccz\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.467776 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.467676 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-serving-certs-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.467776 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.467736 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-metrics-client-ca\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.467867 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.467778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.467867 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.467820 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.467867 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.467847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-federate-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.478583 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.478547 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d"] Oct 30 22:01:48.522447 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.522356 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 22:01:48.522447 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.522381 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 22:01:48.522447 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.522383 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xdcch\"" Oct 30 22:01:48.522447 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.522427 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 22:01:48.522682 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.522465 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 22:01:48.568924 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.568892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.569528 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.568935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.569528 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.569064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ccz\" (UniqueName: \"kubernetes.io/projected/9dce0456-dccb-4e0f-bdbd-978371813dc4-kube-api-access-l4ccz\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.569528 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.569125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-serving-certs-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.569528 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.569187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-metrics-client-ca\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.569528 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.569248 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.569528 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.569292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.569528 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.569323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-federate-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.570061 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.570033 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 22:01:48.570792 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.570758 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-serving-certs-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.570885 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.570857 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-metrics-client-ca\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.572286 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.572259 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.572391 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.572316 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.572450 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.572411 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-federate-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.572897 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.572878 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.580462 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.580436 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.630188 ip-10-0-141-194 kubenswrapper[2583]: I1030 22:01:48.630150 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ccz\" (UniqueName: \"kubernetes.io/projected/9dce0456-dccb-4e0f-bdbd-978371813dc4-kube-api-access-l4ccz\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:48.667153 ip-10-0-141-194 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 22:01:48.679714 ip-10-0-141-194 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 22:01:48.679896 ip-10-0-141-194 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 22:01:48.679950 ip-10-0-141-194 systemd[1]: kubelet.service: Consumed 3.651s CPU time. Oct 30 22:01:48.693901 ip-10-0-141-194 systemd[1]: Starting Kubernetes Kubelet... Oct 30 22:01:48.782892 ip-10-0-141-194 kubenswrapper[5226]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:01:48.782892 ip-10-0-141-194 kubenswrapper[5226]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 22:01:48.782892 ip-10-0-141-194 kubenswrapper[5226]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:01:48.782892 ip-10-0-141-194 kubenswrapper[5226]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 22:01:48.785414 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785306 5226 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:48.785414 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785410 5226 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:48.785414 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785415 5226 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:48.785414 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785419 5226 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785424 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785428 5226 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785435 5226 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785442 5226 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785447 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785451 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785456 5226 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785462 5226 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785475 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785480 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785484 5226 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785489 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785494 5226 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785499 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785503 5226 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785507 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785511 5226 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785515 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:48.785695 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785519 5226 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785523 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785527 5226 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785533 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785537 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785541 5226 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785546 5226 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785550 5226 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785554 5226 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785558 5226 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785563 5226 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785568 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785574 5226 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785578 5226 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785583 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785588 5226 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785592 5226 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785597 5226 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785601 5226 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785606 5226 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:48.786555 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785610 5226 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785616 5226 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785620 5226 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785625 5226 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785630 5226 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785634 5226 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785638 5226 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785643 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785647 5226 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785651 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785656 5226 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785661 5226 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785665 5226 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785670 5226 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785674 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785679 5226 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785683 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785704 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785708 5226 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785713 5226 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:48.787439 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785717 5226 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785721 5226 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785726 5226 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785731 5226 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785735 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785740 5226 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785744 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785753 5226 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785757 5226 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785762 5226 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785766 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785771 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785775 5226 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785782 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785788 5226 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785793 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785797 5226 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785801 5226 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785805 5226 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:48.788321 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785810 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785814 5226 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785819 5226 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785823 5226 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785827 5226 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785831 5226 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785836 5226 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785840 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785845 5226 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785850 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785854 5226 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785859 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785864 5226 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785868 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785872 5226 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785876 5226 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785881 5226 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785886 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785890 5226 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785894 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:48.789156 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785902 5226 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785907 5226 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785911 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785916 5226 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.785920 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787243 5226 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787255 5226 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787260 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787266 5226 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787271 5226 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787276 5226 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787280 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787287 5226 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787294 5226 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787300 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787306 5226 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787311 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787316 5226 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787321 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:48.790039 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787326 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787330 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787335 5226 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787339 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787343 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787348 5226 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787352 5226 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787357 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787361 5226 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787365 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787370 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787374 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787379 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787387 5226 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787391 5226 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787396 5226 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787400 5226 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787405 5226 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787410 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:48.790906 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787414 5226 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787419 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787424 5226 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787429 5226 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787434 5226 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787438 5226 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787443 5226 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787447 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787452 5226 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787457 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787461 5226 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787465 5226 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787470 5226 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787474 5226 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787479 5226 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787483 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787487 5226 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787492 5226 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787496 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787501 5226 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:48.791775 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787505 5226 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787509 5226 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787513 5226 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787518 5226 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787522 5226 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787527 5226 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787532 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787537 5226 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787541 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787545 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787550 5226 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787554 5226 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787558 5226 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787563 5226 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787567 5226 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787571 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787577 5226 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787581 5226 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787586 5226 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787590 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:48.792502 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787594 5226 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787599 5226 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787603 5226 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787608 5226 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787612 5226 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787617 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787621 5226 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787625 5226 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787630 5226 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787634 5226 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787638 5226 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787643 5226 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787647 5226 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787654 5226 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787660 5226 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787664 5226 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787669 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787673 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787677 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:48.793128 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787684 5226 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787689 5226 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787693 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787698 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787703 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787708 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787712 5226 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787717 5226 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787721 5226 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787726 5226 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787732 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787737 5226 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787741 5226 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.787746 5226 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788667 5226 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788678 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788684 5226 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788689 5226 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788695 5226 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:48.793696 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788700 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788706 5226 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788711 5226 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788715 5226 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788719 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788724 5226 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788728 5226 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788733 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788737 5226 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788742 5226 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788747 5226 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788751 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788759 5226 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788764 5226 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788768 5226 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788773 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788777 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788782 5226 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788786 5226 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788790 5226 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:48.794263 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788795 5226 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788800 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788804 5226 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788809 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788814 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788818 5226 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788823 5226 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788827 5226 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788831 5226 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788836 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788840 5226 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788844 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788848 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788853 5226 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788858 5226 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788862 5226 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788866 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788871 5226 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788875 5226 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:48.794924 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788882 5226 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788888 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788893 5226 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788898 5226 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788902 5226 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788908 5226 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788913 5226 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788917 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788921 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788926 5226 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788930 5226 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788934 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788939 5226 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788943 5226 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788950 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788956 5226 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788961 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788966 5226 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788970 5226 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788974 5226 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:48.795629 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788979 5226 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788984 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788988 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.788994 5226 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789000 5226 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789005 5226 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789009 5226 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789013 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789017 5226 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789022 5226 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789026 5226 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789030 5226 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789035 5226 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789039 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789044 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789048 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789052 5226 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789058 5226 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789063 5226 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:48.796281 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789067 5226 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789071 5226 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789075 5226 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789080 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789085 5226 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789089 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789093 5226 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789100 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789104 5226 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789108 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789112 5226 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789118 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789123 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789128 5226 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789134 5226 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789138 5226 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789142 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789147 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789151 5226 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789155 5226 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:48.796883 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789159 5226 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789163 5226 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.789167 5226 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789330 5226 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789353 5226 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789364 5226 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789372 5226 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789379 5226 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789385 5226 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789392 5226 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789406 5226 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789412 5226 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789417 5226 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789423 5226 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789428 5226 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789433 5226 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789438 5226 flags.go:64] FLAG: --cgroup-root="" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789443 5226 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789448 5226 flags.go:64] FLAG: --client-ca-file="" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789454 5226 flags.go:64] FLAG: --cloud-provider="external" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789460 5226 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789466 5226 flags.go:64] FLAG: --cluster-domain="" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789471 5226 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789476 5226 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 22:01:48.797726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789482 5226 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789488 5226 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789496 5226 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789501 5226 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789506 5226 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789511 5226 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789517 5226 flags.go:64] FLAG: --contention-profiling="false" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789523 5226 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789528 5226 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789533 5226 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789538 5226 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789546 5226 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789551 5226 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789556 5226 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789561 5226 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789566 5226 flags.go:64] FLAG: --enable-server="true" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789570 5226 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789578 5226 flags.go:64] FLAG: --event-burst="100" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789586 5226 flags.go:64] FLAG: --event-qps="50" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789591 5226 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789596 5226 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789601 5226 flags.go:64] FLAG: --eviction-hard="" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789607 5226 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789612 5226 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789618 5226 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 22:01:48.798578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789625 5226 flags.go:64] FLAG: --eviction-soft="" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789630 5226 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789637 5226 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789642 5226 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789647 5226 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789652 5226 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789657 5226 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789661 5226 flags.go:64] FLAG: --feature-gates="" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789675 5226 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789680 5226 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789685 5226 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789691 5226 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789696 5226 flags.go:64] FLAG: --healthz-port="10248" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789701 5226 flags.go:64] FLAG: --help="false" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789706 5226 flags.go:64] FLAG: --hostname-override="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789712 5226 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789717 5226 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789722 5226 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789728 5226 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789734 5226 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789739 5226 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789744 5226 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789749 5226 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 22:01:48.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789754 5226 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789759 5226 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789767 5226 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789771 5226 flags.go:64] FLAG: --kube-reserved="" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789776 5226 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789781 5226 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789786 5226 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789791 5226 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789796 5226 flags.go:64] FLAG: --lock-file="" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789801 5226 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789806 5226 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789814 5226 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789823 5226 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789827 5226 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789833 5226 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789838 5226 flags.go:64] FLAG: --logging-format="text" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789843 5226 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789849 5226 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789854 5226 flags.go:64] FLAG: --manifest-url="" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789860 5226 flags.go:64] FLAG: --manifest-url-header="" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789867 5226 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789872 5226 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789879 5226 flags.go:64] FLAG: --max-pods="110" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789884 5226 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789889 5226 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 22:01:48.800105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789894 5226 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789899 5226 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789904 5226 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789909 5226 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789913 5226 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789925 5226 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789930 5226 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789935 5226 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789940 5226 flags.go:64] FLAG: --pod-cidr="" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789947 5226 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789952 5226 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789957 5226 flags.go:64] FLAG: --pods-per-core="0" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789962 5226 flags.go:64] FLAG: --port="10250" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789968 5226 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789973 5226 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-096dc96b296ec6c66" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789978 5226 flags.go:64] FLAG: --qos-reserved="" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789983 5226 flags.go:64] FLAG: --read-only-port="10255" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789988 5226 flags.go:64] FLAG: --register-node="true" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.789995 5226 flags.go:64] FLAG: --register-with-taints="" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790002 5226 flags.go:64] FLAG: --registry-burst="10" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790007 5226 flags.go:64] FLAG: --registry-qps="5" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790012 5226 flags.go:64] FLAG: --reserved-cpus="" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790017 5226 flags.go:64] FLAG: --reserved-memory="" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790024 5226 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790030 5226 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790034 5226 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 22:01:48.801025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790040 5226 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790045 5226 flags.go:64] FLAG: --runonce="false" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790049 5226 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790055 5226 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790060 5226 flags.go:64] FLAG: --seccomp-default="false" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790065 5226 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790070 5226 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790075 5226 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790081 5226 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790086 5226 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790091 5226 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790095 5226 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790100 5226 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790105 5226 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790110 5226 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790117 5226 flags.go:64] FLAG: --system-cgroups="" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790123 5226 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790132 5226 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790137 5226 flags.go:64] FLAG: --tls-cert-file="" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790142 5226 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790148 5226 flags.go:64] FLAG: --tls-min-version="" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790153 5226 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790158 5226 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790162 5226 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790170 5226 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 22:01:48.801820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790175 5226 flags.go:64] FLAG: --v="2" Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790182 5226 flags.go:64] FLAG: --version="false" Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790189 5226 flags.go:64] FLAG: --vmodule="" Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790196 5226 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.790218 5226 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790481 5226 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790490 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790495 5226 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790501 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790506 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790510 5226 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790515 5226 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790519 5226 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790524 5226 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790528 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790533 5226 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790537 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790542 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790547 5226 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790551 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:48.802721 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790558 5226 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790564 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790572 5226 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790577 5226 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790582 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790588 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790593 5226 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790598 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790603 5226 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790608 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790612 5226 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790619 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790624 5226 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790628 5226 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790633 5226 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790637 5226 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790643 5226 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790647 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790652 5226 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:48.803583 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790656 5226 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790661 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790667 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790671 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790676 5226 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790680 5226 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790685 5226 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790690 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790694 5226 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790699 5226 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790703 5226 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790707 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790712 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790717 5226 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790722 5226 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790727 5226 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790732 5226 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790736 5226 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790740 5226 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790745 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:48.804185 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790750 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790755 5226 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790759 5226 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790766 5226 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790774 5226 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790779 5226 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790784 5226 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790789 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790794 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790799 5226 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790803 5226 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790808 5226 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790812 5226 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790816 5226 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790821 5226 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790826 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790831 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790835 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790840 5226 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:48.804931 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790844 5226 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790849 5226 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790853 5226 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790857 5226 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790861 5226 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790866 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790870 5226 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790875 5226 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790881 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790886 5226 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790890 5226 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790894 5226 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790899 5226 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790903 5226 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790908 5226 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790912 5226 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790916 5226 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790924 5226 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790929 5226 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790933 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:48.805755 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790938 5226 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790943 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790948 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790952 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790957 5226 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790962 5226 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790966 5226 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790971 5226 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790975 5226 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790980 5226 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790985 5226 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790990 5226 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.790994 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.791040 5226 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791098 5226 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791105 5226 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:48.806518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791110 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791115 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791121 5226 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791125 5226 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791131 5226 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791136 5226 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791142 5226 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791147 5226 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791151 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791156 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791161 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791165 5226 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791170 5226 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791174 5226 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791181 5226 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791185 5226 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791189 5226 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791194 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791199 5226 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791220 5226 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:48.807020 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791224 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791229 5226 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791233 5226 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791238 5226 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791242 5226 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791246 5226 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791252 5226 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791257 5226 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791262 5226 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791267 5226 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791271 5226 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791276 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791281 5226 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791285 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791290 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791294 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791299 5226 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791303 5226 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791309 5226 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791313 5226 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:48.807790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791318 5226 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791322 5226 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791327 5226 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791331 5226 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791336 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791340 5226 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791346 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791351 5226 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791356 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791361 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791365 5226 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791370 5226 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791374 5226 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791379 5226 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791384 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791388 5226 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791394 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791399 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791403 5226 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:48.808564 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791409 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791415 5226 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791421 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791426 5226 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791431 5226 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791435 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791440 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791444 5226 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791449 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791453 5226 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791458 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791462 5226 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791467 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791471 5226 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791475 5226 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791480 5226 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791484 5226 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791491 5226 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791495 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:48.809309 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791502 5226 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791506 5226 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791511 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791516 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791520 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791525 5226 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791529 5226 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791534 5226 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791539 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791543 5226 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791548 5226 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791552 5226 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791557 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791561 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791567 5226 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791571 5226 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791576 5226 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791580 5226 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791584 5226 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791589 5226 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:48.809981 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791593 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791598 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791603 5226 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791608 5226 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791616 5226 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.791623 5226 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.797135 5226 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.797158 5226 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.797187 5226 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.797195 5226 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797496 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797506 5226 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797511 5226 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797516 5226 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797521 5226 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797526 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797531 5226 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797541 5226 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797546 5226 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:48.810687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797551 5226 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797555 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797559 5226 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797564 5226 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797569 5226 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797573 5226 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797578 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797583 5226 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797588 5226 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797592 5226 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797597 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797606 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797610 5226 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797615 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797619 5226 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797623 5226 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797628 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797632 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797637 5226 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797642 5226 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:48.811367 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797649 5226 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797654 5226 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797658 5226 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797667 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797671 5226 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797676 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797681 5226 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797685 5226 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797689 5226 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797694 5226 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797699 5226 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797703 5226 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797708 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797712 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797716 5226 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797721 5226 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797730 5226 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797735 5226 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797740 5226 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797745 5226 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:48.812012 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797750 5226 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797754 5226 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797759 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797764 5226 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797768 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797774 5226 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797778 5226 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797783 5226 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797788 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797798 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797802 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797807 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797812 5226 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797842 5226 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797847 5226 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797852 5226 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797856 5226 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797860 5226 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797864 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:48.812720 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797869 5226 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797873 5226 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797883 5226 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797887 5226 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797894 5226 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797901 5226 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797906 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797912 5226 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797917 5226 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797922 5226 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797925 5226 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797928 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.797931 5226 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798179 5226 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798252 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798510 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798522 5226 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798528 5226 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798534 5226 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:48.813416 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798539 5226 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798543 5226 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798548 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798552 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798556 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798561 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798565 5226 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798570 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798574 5226 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798578 5226 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798584 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798589 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798594 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798599 5226 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798604 5226 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798609 5226 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798613 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798620 5226 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:48.814023 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798629 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.798679 5226 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798960 5226 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798969 5226 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798974 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798979 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798983 5226 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798988 5226 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798992 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.798997 5226 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799002 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799007 5226 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799013 5226 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799019 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799024 5226 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:48.814638 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799029 5226 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799033 5226 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799037 5226 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799042 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799046 5226 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799050 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799054 5226 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799059 5226 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799064 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799068 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799073 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799077 5226 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799081 5226 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799085 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799090 5226 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799095 5226 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799099 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799104 5226 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799108 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:48.815157 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799113 5226 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799118 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799122 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799126 5226 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799131 5226 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799136 5226 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799141 5226 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799146 5226 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799152 5226 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799156 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799161 5226 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799165 5226 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799169 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799174 5226 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799178 5226 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799183 5226 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799187 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799192 5226 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799197 5226 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799214 5226 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:48.815822 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799220 5226 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799224 5226 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799228 5226 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799233 5226 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799237 5226 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799242 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799247 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799251 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799256 5226 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799260 5226 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799264 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799269 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799274 5226 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799278 5226 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799283 5226 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799287 5226 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799291 5226 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799296 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799300 5226 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:48.816584 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799311 5226 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799315 5226 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799320 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799325 5226 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799329 5226 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799333 5226 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799338 5226 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799342 5226 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799346 5226 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799350 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799355 5226 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799359 5226 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799363 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799368 5226 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799372 5226 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799377 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799381 5226 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799385 5226 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799389 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799394 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:48.817195 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799398 5226 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799403 5226 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799408 5226 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799412 5226 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799417 5226 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799421 5226 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799426 5226 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799430 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799434 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799439 5226 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799443 5226 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799447 5226 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799453 5226 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799462 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799467 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:48.817901 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.799507 5226 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799560 5226 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799569 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799575 5226 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799580 5226 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799584 5226 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799589 5226 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799593 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799597 5226 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799603 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799607 5226 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799612 5226 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799616 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799621 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799625 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799629 5226 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799634 5226 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799638 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799642 5226 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 22:01:48.818450 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799647 5226 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799652 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799659 5226 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799664 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799668 5226 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799672 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799677 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799681 5226 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799685 5226 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799689 5226 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799694 5226 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799698 5226 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799704 5226 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799708 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799713 5226 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799717 5226 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799721 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799726 5226 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799731 5226 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799735 5226 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 22:01:48.819122 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799740 5226 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799745 5226 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799750 5226 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799754 5226 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799758 5226 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799763 5226 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799768 5226 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799772 5226 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799776 5226 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799781 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799785 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799790 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799794 5226 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799798 5226 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799805 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799810 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799814 5226 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799819 5226 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799823 5226 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799827 5226 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 22:01:48.819899 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799832 5226 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799836 5226 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799841 5226 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799845 5226 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799849 5226 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799855 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799860 5226 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799865 5226 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799869 5226 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799873 5226 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799878 5226 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799884 5226 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799888 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799893 5226 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799897 5226 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799901 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799906 5226 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799910 5226 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799915 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799919 5226 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 22:01:48.820625 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799923 5226 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799928 5226 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799932 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799937 5226 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799941 5226 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799945 5226 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799951 5226 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799956 5226 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799960 5226 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799965 5226 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799969 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799973 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799978 5226 feature_gate.go:386] unrecognized feature gate: Example Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799982 5226 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799986 5226 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.799993 5226 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800000 5226 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800005 5226 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800012 5226 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 22:01:48.821342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800016 5226 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800022 5226 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800027 5226 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800031 5226 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800036 5226 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800040 5226 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800044 5226 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800049 5226 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:48.800055 5226 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.800151 5226 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.801872 5226 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.801950 5226 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.802247 5226 server.go:1014] "Starting client certificate rotation" Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.802442 5226 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.802504 5226 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:55:37 +0000 UTC" deadline="2028-08-17 16:04:07.071736588 +0000 UTC" Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.802581 5226 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15762h2m18.269159009s" Oct 30 22:01:48.821946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.802831 5226 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 22:01:48.822493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.804394 5226 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 22:01:48.822493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.806181 5226 log.go:25] "Validated CRI v1 runtime API" Oct 30 22:01:48.822493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.808513 5226 log.go:25] "Validated CRI v1 image API" Oct 30 22:01:48.822493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.809605 5226 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 22:01:48.822493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.813588 5226 fs.go:135] Filesystem UUIDs: map[24c5e27f-3136-491c-8c5c-45ce9470f807:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f04bdf9d-7ee1-42a3-adeb-ddb1bfa1499d:/dev/nvme0n1p4] Oct 30 22:01:48.822789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.813620 5226 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02c601c74d0744c0328576bd161bc5534ed6d4c6f9dca96aa9cf17f70399debf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02c601c74d0744c0328576bd161bc5534ed6d4c6f9dca96aa9cf17f70399debf/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0679eeb4ff57c90459211f3d32808d284ca564b9d34c24c8e09552cd4e2007d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0679eeb4ff57c90459211f3d32808d284ca564b9d34c24c8e09552cd4e2007d5/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08dad873475d087043c9cf3f64513901ba39638bff4b6e9205dbe8657deb1e15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08dad873475d087043c9cf3f64513901ba39638bff4b6e9205dbe8657deb1e15/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0cfbc5e5215ff32bb6d5854f5dcd92f3efc54b9be9172cde3ab1344e2176025c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0cfbc5e5215ff32bb6d5854f5dcd92f3efc54b9be9172cde3ab1344e2176025c/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22e292e20416440ade309fdfa2e737c5af722d3d10d76f629bb6bfd3b07df45f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22e292e20416440ade309fdfa2e737c5af722d3d10d76f629bb6bfd3b07df45f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c319c110faae66e896cb9bb8e246decbe483c4857bbc3e6f6ea12e5916b9af7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c319c110faae66e896cb9bb8e246decbe483c4857bbc3e6f6ea12e5916b9af7/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41c1ffe91871d12bf833318f37eab174b36f86bf4002ebe9dd45ae4fa1d24983/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41c1ffe91871d12bf833318f37eab174b36f86bf4002ebe9dd45ae4fa1d24983/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4642ae4eba01be1299c42eb440d862b9af8b1a2089e6d7b0f1ada23dedd6c357/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4642ae4eba01be1299c42eb440d862b9af8b1a2089e6d7b0f1ada23dedd6c357/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f7fd4d0a8e58793dd57e2366d805e66f5610b2adebbcc7a33cc1e6af6779994/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f7fd4d0a8e58793dd57e2366d805e66f5610b2adebbcc7a33cc1e6af6779994/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7552fcdfe513c4d4a5e6b2333faf9987c0c4359fa2ca9681c30eaa331d192049/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7552fcdfe513c4d4a5e6b2333faf9987c0c4359fa2ca9681c30eaa331d192049/userdata/shm major:0 minor:321 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ae20dfd94b056b365e689e2cb06a6a7b079ae3d37f3f2cfb3949d55fa19a987/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ae20dfd94b056b365e689e2cb06a6a7b079ae3d37f3f2cfb3949d55fa19a987/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ca9772597ceae78fddadb74ae19168082d214b4a2ae820a325ccddf54578e52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ca9772597ceae78fddadb74ae19168082d214b4a2ae820a325ccddf54578e52/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cd6cd6d89bf397e6c645743679bb13189d8ee69d1ca5fdc1536d52a56b37d93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cd6cd6d89bf397e6c645743679bb13189d8ee69d1ca5fdc1536d52a56b37d93/userdata/shm major:0 minor:280 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8eaa2d97e2fb5b7f114b775d094f69c7bec98c2e3e155f90d1dc8ce7cffc5182/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8eaa2d97e2fb5b7f114b775d094f69c7bec98c2e3e155f90d1dc8ce7cffc5182/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8097e11515b58ec89c2a45e643c054bdd57dad923ba8908b96adcfb5fdf41a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8097e11515b58ec89c2a45e643c054bdd57dad923ba8908b96adcfb5fdf41a1/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae1048f517c07efa6ea8af000c9ba83fd79df54c9c4b6e01e9006254d50191fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae1048f517c07efa6ea8af000c9ba83fd79df54c9c4b6e01e9006254d50191fb/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af1c410213e019092d196caca5c15b78b8f9190741c49bc0f14d0b3a18dc9ee5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af1c410213e019092d196caca5c15b78b8f9190741c49bc0f14d0b3a18dc9ee5/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c768f6881cdb1936971434cd7c791005a8928c6e83cbac8b7bfd9bda816289aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c768f6881cdb1936971434cd7c791005a8928c6e83cbac8b7bfd9bda816289aa/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4068e99b303b1e9e7c590611c43a0fdcb2c0e02998f52454ec7d1e61417f808/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4068e99b303b1e9e7c590611c43a0fdcb2c0e02998f52454ec7d1e61417f808/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb36f3a37ce2fb052329620d5a8990bcab49c8a6d8a76e7f7db8bfd778180a3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb36f3a37ce2fb052329620d5a8990bcab49c8a6d8a76e7f7db8bfd778180a3d/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed1d70f775129e8c5081b6c9c35aeaca64eff12ade600d9b2540fc739b58901e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed1d70f775129e8c5081b6c9c35aeaca64eff12ade600d9b2540fc739b58901e/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/089436e9-8607-4901-9b23-8e0512aacc3c/volumes/kubernetes.io~projected/kube-api-access-lgvlp:{mountpoint:/var/lib/kubelet/pods/089436e9-8607-4901-9b23-8e0512aacc3c/volumes/kubernetes.io~projected/kube-api-access-lgvlp major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/089436e9-8607-4901-9b23-8e0512aacc3c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/089436e9-8607-4901-9b23-8e0512aacc3c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~projected/kube-api-access-d9bbq:{mountpoint:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~projected/kube-api-access-d9bbq major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volumes/kubernetes.io~projected/kube-api-access-5gpgg:{mountpoint:/var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volumes/kubernetes.io~projected/kube-api-access-5gpgg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b8fc6dd-36fe-4714-bcda-1863280a8324/volumes/kubernetes.io~projected/kube-api-access-6frbn:{mountpoint:/var/lib/kubelet/pods/1b8fc6dd-36fe-4714-bcda-1863280a8324/volumes/kubernetes.io~projected/kube-api-access-6frbn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20b0ee9f-f99d-472a-a1d8-df44248d9aec/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/20b0ee9f-f99d-472a-a1d8-df44248d9aec/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2675c1b0-71d1-40b8-b78c-5ef4a59389d7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2675c1b0-71d1-40b8-b78c-5ef4a59389d7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~projected/kube-api-access-wpbrr:{mountpoint:/var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~projected/kube-api-access-wpbrr major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~projected/kube-api-access-5bhtx:{mountpoint:/var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~projected/kube-api-access-5bhtx major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/588b62a8-9745-4e5a-97dd-f8c451197018/volumes/kubernetes.io~projected/kube-api-access-gzgj8:{mountpoint:/var/lib/kubelet/pods/588b62a8-9745-4e5a-97dd-f8c451197018/volumes/kubernetes.io~projected/kube-api-access-gzgj8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/kube-api-access-2l64c:{mountpoint:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/kube-api-access-2l64c major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/registry-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28/volumes/kubernetes.io~projected/kube-api-access-8kxtv:{mountpoint:/var/lib/kubelet/pods/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28/volumes/kubernetes.io~projected/kube-api-access-8kxtv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~projected/kube-api-access-dd8k8:{mountpoint:/var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~projected/kube-api-access-dd8k8 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d531260-0f02-4b21-af0a-ec862fca540d/volumes/kubernetes.io~projected/kube-api-access-wsdpq:{mountpoint:/var/lib/kubelet/pods/8d531260-0f02-4b21-af0a-ec862fca540d/volumes/kubernetes.io~projected/kube-api-access-wsdpq major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d531260-0f02-4b21-af0a-ec862fca540d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8d531260-0f02-4b21-af0a-ec862fca540d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f434211-8bee-4801-9c8e-39f0bb0e71d6/volumes/kubernetes.io~projected/kube-api-access-445ts:{mountpoint:/var/lib/kubelet/pods/8f434211-8bee-4801-9c8e-39f0bb0e71d6/volumes/kubernetes.io~projected/kube-api-access-445ts major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94ca8590-27e5-497d-9bb8-cb5041ceb267/volumes/kubernetes.io~projected/kube-api-access-g8jq7:{mountpoint:/var/lib/kubelet/pods/94ca8590-27e5-497d-9bb8-cb5041ceb267/volumes/kubernetes.io~projected/kube-api-access-g8jq7 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~projected/kube-api-access-l4ccz:{mountpoint:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~projected/kube-api-access-l4ccz major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3b54602-8b76-49b7-b295-42969da3c4fd/volumes/kubernetes.io~projected/kube-api-access-nkwvd:{mountpoint:/var/lib/kubelet/pods/b3b54602-8b76-49b7-b295-42969da3c4fd/volumes/kubernetes.io~projected/kube-api-access-nkwvd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba98bdc2-2e27-4499-8182-4abf75b94a23/volumes/kubernetes.io~projected/kube-api-access-kppgx:{mountpoint:/var/lib/kubelet/pods/ba98bdc2-2e27-4499-8182-4abf75b94a23/volumes/kubernetes.io~projected/kube-api-access-kppgx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba98bdc2-2e27-4499-8182-4abf75b94a23/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ba98bdc2-2e27-4499-8182-4abf75b94a23/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c986bec8-11a1-4c93-86fa-40b486199598/volumes/kubernetes.io~projected/kube-api-access-vfzp9:{mountpoint:/var/lib/kubelet/pods/c986bec8-11a1-4c93-86fa-40b486199598/volumes/kubernetes.io~projected/kube-api-access-vfzp9 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfbc34e7-48ae-456a-ad03-408b46e20053/volumes/kubernetes.io~projected/kube-api-access-hcv94:{mountpoint:/var/lib/kubelet/pods/cfbc34e7-48ae-456a-ad03-408b46e20053/volumes/kubernetes.io~projected/kube-api-access-hcv94 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfbc34e7-48ae-456a-ad03-408b46e20053/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/cfbc34e7-48ae-456a-ad03-408b46e20053/volumes/kubernetes.io~secret/metrics-certs major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~projected/kube-api-access-gprrl:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~projected/kube-api-access-gprrl major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/config-volume major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/web-config major:0 minor:310 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/31818de58e563c391a9c7061f514be5b192d52d0faa86e028eb10ea21340d00b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5cb615b2c000db48e9490693d31ea4bd89a5bd96c58c9ae7954c88ba90ac5002/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/08959000f1202880eb782d7a78acde25db3c1c86f3126ac7c46f7ef8e9bf4653/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ab6f2de50b0d087bee0a08d310f50444a504c453f627604d6435f591831184de/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/f713a335ea97595953cbc30249397d91bc801adb8ff7733b109e01a2b192e5a7/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/763bb239d12ae0eb0d07c86f109ea1aa9123d374f9308285f93c9b69d7c000f4/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/13f4ead7bd17b9cc8a4a05fe1d54e0daa64441ca45d0eb0a8314c685af9f0bcb/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/49ac51d733c4653074debde21aaf4c8ccbf04b021e2db57841fd674ed2da7c83/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4fa2e5439d2c07f5dce02123a50b32ee94824517edecb50724e190433c024f01/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/382be399a0f89cba90de8fa42090d51df87a2c2b227f5b6baf48d4be5af98dbb/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/fe9e6615c83725fd2f7cf901e0250692ac2660f22c4e0ec91871b0e530c4b590/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/625e5b29eaea11957b4f7ed79e388880bf1161d6019de04d7f35e647a4d142a6/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6fb19370b54ff18a4a2b6d1660fa0bf96b7b61c1f2e99e1e14005732aa7e9bd4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4948b64eee68ede9107990bab17d482cba4a0527256c9df07e9d8b067f84e437/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/76bbe4ac2df0fd0e26fd8761d39e2efcff9660c6fda7753677d283068c09c760/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/b7827731affbab0bba4577233f7953d87f4c5f062a89360f0a07538b15660a2a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8d2d633d30325a3f3c9935bb7bea867ba1b2ea5ecce257b0f7694cd23e96431d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cc739a1e5dadb4700f550f95878547e487e6cfdf74d10f222a6a6ed88ccc52ff/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0e00c978896691e35b9100fbeb6a9eff69e3593855b99af8e6581ee1e33ea031/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f197d8f83880448a4df82f98a80597b347be258f2c33cd3a83cd6b8ec5dadc66/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f1ece7720497cd59bc784698662ccab2a3618530efac976cdaef1e6d495779f6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/b084c6e3a8caa9c2da0324db375edcec49b8b72781f7d8a49ae559c083a1ef49/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/77d1d54033208993c2ea06f511e3ccec5710618a36b0c000f8afb08d68a749f7/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/87b00b88e63ec8848d2dab7cb36e756d5d9a174b062967543700127b8005a2f0/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ff9ef4b82e3d89dd2600363c56c52673a1c3e675198c347701e293970717acaa/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/55a35e65d81a72c263c6979e31507815e8da7fb000ccc02feee37c1e586b0821/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/eac6257ec119052f87803aa4adeb70fde9ecc1dcfbaae4fb392d8cbf31f0173a/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/c0c474ea1767f152d7cad7f6ba2d99c3687f44b2e978b6e920b0cb187500d2c2/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/9a20f65d285ca48232943ef2c2a48d321a994dc959dc9277cf87ea7d4d37c67b/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4c3d27c2f2b0a0e2363c2ccbf10bcb6b016d8209d3a15087ad1a653104724d0a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/5da9a39fefdfa879d2d02e2e9bc5161a25edb2a43600df9b9299aeca70bd24eb/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/2eae049badc81346446727eb105b5276fe3b42844e7ed4f07dc7af02e014f2fc/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/90020b0a8591bd45182441b4b2615c297ffb269497ad3e879a1983ffe6e58593/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/724d191ae760de91bbd8e6d587115a0b02a6f032e8e960dafd9721f8fe1e00a0/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/04e2016cd8648104e68f2332228b81cba38cc3010a0a47e54b3746802a2595cc/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/61860eadc4446a7ef448896e778352826b054fa40450ad25159fdc960c7c2894/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/8392ad67fc5e1bccfdf76031d4121c714f90ac2892d10a98200f9347aab628e5/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/ffe52db4459df868414b0c249202f0e71418c79228e1699c201feed6779abc57/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/8ceb89f30801de85d3fbf04cb3b6ef7de21d3c611b89eefc6db3f460505cd6d7/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/139aafe544615779f89e7b92595a200798a6e6655abe05fa804901ed52a6e6ed/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/b0c488ac115a076c0a016338564b00481bd296cda001c098ebc436f75bfc80a3/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/5800898a22c51bc57e058eea8137ccdcfae3eae55a23a95deeb16205bd5bcc6c/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/31a0924c9e5a38d5348980caf797f85aa94567f4cddac533cba4417cc5e309e5/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/2e25a323c9c56e19eeec30826097f17f4040ba8e9bd24baf710675d25bcf23a2/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-46:{mountpoint:/var/lib/containers/storage/overlay/a5bad2e8d06dfc626f9c89253959851878f8dbf6fdbb1795fcf7c430e4376837/merged major:0 minor:46 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/4e5f22a8bccf1f299d33379cbcd0b7cdd1ca37b84b02a92cdec007b42a064d01/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/ca676edd8c126ea4c808f4b42887adc3118e8ec023bbcac5d3815d70cf75fa77/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5ffc494d01e6022bbc557d8df1cbcaf58dcc04becb0484dafe7943823a6de48a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9fcf1ce6945b153c5e80d9844015615e9e87bc629fc061ae98c8b33d7716ab38/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/355722ffb6867ef15ad859dcd26f934f87cc93551167d50a648cb312bcc91547/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c043a5a88cd015d689d69d8eacb18e0deebc82716b17094046912011cfd9ac0d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/c8517762a6daaf809073708eacfcde9dba8680b6a4fe419d9a2f813e7965f161/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/8cd94f4b6ceb4582c027c75026ab1998e178bc4c63a7bc70ee982be2fbcab08d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/4ffa44c01cbf312e5b7a32280ece33925d2bf8d567a16069f2195c35727b032d/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/a9252a54aac6d569c132f57f83464eac8e77504fa9b971fcdb509786d5e9775a/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 22:01:48.828681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.828010 5226 manager.go:217] Machine: {Timestamp:2026-10-30 22:01:48.827162247 +0000 UTC m=+0.084503949 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098971 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c3ab4f3fbce05a8e4803debe0cffe SystemUUID:ec2c3ab4-f3fb-ce05-a8e4-803debe0cffe BootID:fd8771b6-8815-4c05-8996-e69a1525746d Filesystems:[{Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c319c110faae66e896cb9bb8e246decbe483c4857bbc3e6f6ea12e5916b9af7/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfbc34e7-48ae-456a-ad03-408b46e20053/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:273 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1b8fc6dd-36fe-4714-bcda-1863280a8324/volumes/kubernetes.io~projected/kube-api-access-6frbn DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:305 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/089436e9-8607-4901-9b23-8e0512aacc3c/volumes/kubernetes.io~projected/kube-api-access-lgvlp DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae1048f517c07efa6ea8af000c9ba83fd79df54c9c4b6e01e9006254d50191fb/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2675c1b0-71d1-40b8-b78c-5ef4a59389d7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af1c410213e019092d196caca5c15b78b8f9190741c49bc0f14d0b3a18dc9ee5/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ca9772597ceae78fddadb74ae19168082d214b4a2ae820a325ccddf54578e52/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/089436e9-8607-4901-9b23-8e0512aacc3c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:242 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:306 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~projected/kube-api-access-wpbrr DeviceMajor:0 DeviceMinor:342 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0679eeb4ff57c90459211f3d32808d284ca564b9d34c24c8e09552cd4e2007d5/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c986bec8-11a1-4c93-86fa-40b486199598/volumes/kubernetes.io~projected/kube-api-access-vfzp9 DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4642ae4eba01be1299c42eb440d862b9af8b1a2089e6d7b0f1ada23dedd6c357/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4068e99b303b1e9e7c590611c43a0fdcb2c0e02998f52454ec7d1e61417f808/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~projected/kube-api-access-l4ccz DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:327 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:349 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-46 DeviceMajor:0 DeviceMinor:46 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:250 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cd6cd6d89bf397e6c645743679bb13189d8ee69d1ca5fdc1536d52a56b37d93/userdata/shm DeviceMajor:0 DeviceMinor:280 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08dad873475d087043c9cf3f64513901ba39638bff4b6e9205dbe8657deb1e15/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8eaa2d97e2fb5b7f114b775d094f69c7bec98c2e3e155f90d1dc8ce7cffc5182/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/94ca8590-27e5-497d-9bb8-cb5041ceb267/volumes/kubernetes.io~projected/kube-api-access-g8jq7 DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:323 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d531260-0f02-4b21-af0a-ec862fca540d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20b0ee9f-f99d-472a-a1d8-df44248d9aec/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cfbc34e7-48ae-456a-ad03-408b46e20053/volumes/kubernetes.io~projected/kube-api-access-hcv94 DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:307 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:345 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a72d78e-c521-4dba-b551-5970a7505245/volumes/kubernetes.io~projected/kube-api-access-5gpgg DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10f4cba6-57dc-4397-a602-21a3bad21b82/volumes/kubernetes.io~projected/kube-api-access-d9bbq DeviceMajor:0 DeviceMinor:332 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed1d70f775129e8c5081b6c9c35aeaca64eff12ade600d9b2540fc739b58901e/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:254 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~projected/kube-api-access-gprrl DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/588b62a8-9745-4e5a-97dd-f8c451197018/volumes/kubernetes.io~projected/kube-api-access-gzgj8 DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b3b54602-8b76-49b7-b295-42969da3c4fd/volumes/kubernetes.io~projected/kube-api-access-nkwvd DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70/volumes/kubernetes.io~projected/kube-api-access-2l64c DeviceMajor:0 DeviceMinor:253 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb36f3a37ce2fb052329620d5a8990bcab49c8a6d8a76e7f7db8bfd778180a3d/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9dce0456-dccb-4e0f-bdbd-978371813dc4/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8097e11515b58ec89c2a45e643c054bdd57dad923ba8908b96adcfb5fdf41a1/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f522003-5ca0-45d8-b32e-b52db6916aca/volumes/kubernetes.io~projected/kube-api-access-5bhtx DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ae20dfd94b056b365e689e2cb06a6a7b079ae3d37f3f2cfb3949d55fa19a987/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:312 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3058b5f8-ae0e-4193-b4dd-6d4abace3282/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d531260-0f02-4b21-af0a-ec862fca540d/volumes/kubernetes.io~projected/kube-api-access-wsdpq DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7552fcdfe513c4d4a5e6b2333faf9987c0c4359fa2ca9681c30eaa331d192049/userdata/shm DeviceMajor:0 DeviceMinor:321 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8f434211-8bee-4801-9c8e-39f0bb0e71d6/volumes/kubernetes.io~projected/kube-api-access-445ts DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8420c80f-804e-468e-8df2-fb1ec145d318/volumes/kubernetes.io~projected/kube-api-access-dd8k8 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02c601c74d0744c0328576bd161bc5534ed6d4c6f9dca96aa9cf17f70399debf/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0cfbc5e5215ff32bb6d5854f5dcd92f3efc54b9be9172cde3ab1344e2176025c/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f7fd4d0a8e58793dd57e2366d805e66f5610b2adebbcc7a33cc1e6af6779994/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28/volumes/kubernetes.io~projected/kube-api-access-8kxtv DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba98bdc2-2e27-4499-8182-4abf75b94a23/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:311 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41c1ffe91871d12bf833318f37eab174b36f86bf4002ebe9dd45ae4fa1d24983/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ba98bdc2-2e27-4499-8182-4abf75b94a23/volumes/kubernetes.io~projected/kube-api-access-kppgx DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22e292e20416440ade309fdfa2e737c5af722d3d10d76f629bb6bfd3b07df45f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c768f6881cdb1936971434cd7c791005a8928c6e83cbac8b7bfd9bda816289aa/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02c601c74d0744c MacAddress:2e:f4:e1:e2:3a:5c Speed:10000 Mtu:8901} {Name:7552fcdfe513c4d MacAddress:32:02:42:b5:c8:5b Speed:10000 Mtu:8901} {Name:7ae20dfd94b056b MacAddress:2a:9c:99:10:bb:3e Speed:10000 Mtu:8901} {Name:8ca9772597ceae7 MacAddress:86:18:b6:b4:6f:27 Speed:10000 Mtu:8901} {Name:8cd6cd6d89bf397 MacAddress:2e:eb:46:b9:dc:61 Speed:10000 Mtu:8901} {Name:8eaa2d97e2fb5b7 MacAddress:7e:b4:84:4e:ab:ab Speed:10000 Mtu:8901} {Name:a8097e11515b58e MacAddress:3e:c4:4c:ea:40:38 Speed:10000 Mtu:8901} {Name:ae1048f517c07ef MacAddress:92:47:5b:4e:ea:d4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d3:bd:9f:e3 Speed:0 Mtu:9001} {Name:br-int MacAddress:72:37:df:5f:aa:56 Speed:0 Mtu:8901} {Name:eb36f3a37ce2fb0 MacAddress:f6:19:b3:46:dc:e3 Speed:10000 Mtu:8901} {Name:ed1d70f775129e8 MacAddress:86:72:b2:d0:9e:4f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d3:bd:9f:e3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8a:4b:5d:3f:4b:ea Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fe:2b:51:d1:3c:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 22:01:48.829052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.828695 5226 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 22:01:48.829052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.828770 5226 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 22:01:48.829199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829165 5226 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 22:01:48.829411 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829217 5226 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-194.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 22:01:48.829507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829416 5226 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 22:01:48.829507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829428 5226 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 22:01:48.829507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829448 5226 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 22:01:48.829507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829467 5226 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 22:01:48.829657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829630 5226 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 22:01:48.829882 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829861 5226 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 22:01:48.829998 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.829984 5226 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 22:01:48.830083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.830032 5226 kubelet.go:493] "Attempting to sync node with API server" Oct 30 22:01:48.830083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.830047 5226 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 22:01:48.830083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.830078 5226 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 22:01:48.830332 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.830095 5226 kubelet.go:405] "Adding apiserver pod source" Oct 30 22:01:48.830332 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.830111 5226 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 22:01:48.831168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.831148 5226 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 22:01:48.831168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.831166 5226 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 22:01:48.831784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.831737 5226 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 22:01:48.843020 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.842984 5226 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 22:01:48.843020 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843004 5226 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 22:01:48.843334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843177 5226 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 22:01:48.843516 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843499 5226 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 22:01:48.843594 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843542 5226 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 22:01:48.843727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843710 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 22:01:48.843806 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843735 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 22:01:48.843806 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843770 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 22:01:48.843806 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843782 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 22:01:48.843806 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843791 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 22:01:48.843806 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843799 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 22:01:48.844042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843810 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 22:01:48.844042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843819 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 22:01:48.844042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843830 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 22:01:48.844042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843839 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 22:01:48.844042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843862 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 22:01:48.844042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843879 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 22:01:48.844042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843918 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 22:01:48.844042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.843928 5226 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 22:01:48.848540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.848519 5226 server.go:1263] "Started kubelet" Oct 30 22:01:48.848694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.848633 5226 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 22:01:48.848743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.848648 5226 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 22:01:48.848784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.848765 5226 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 22:01:48.849100 ip-10-0-141-194 systemd[1]: Started Kubernetes Kubelet. Oct 30 22:01:48.849820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.849302 5226 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 22:01:48.850306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.850285 5226 server.go:317] "Adding debug handlers to kubelet server" Oct 30 22:01:48.856749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.856719 5226 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 22:01:48.857778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.857643 5226 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 22:01:48.858463 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.858368 5226 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 22:01:48.858463 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.858392 5226 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 22:01:48.858652 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.858636 5226 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 22:01:48.858772 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:48.858707 5226 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" Oct 30 22:01:48.859980 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.859935 5226 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:55:38 +0000 UTC" deadline="2028-07-15 08:03:34.8694875 +0000 UTC" Oct 30 22:01:48.859980 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.859974 5226 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14962h1m46.009516473s" Oct 30 22:01:48.861999 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.861973 5226 factory.go:55] Registering systemd factory Oct 30 22:01:48.861999 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.862004 5226 factory.go:223] Registration of the systemd container factory successfully Oct 30 22:01:48.862289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.862241 5226 factory.go:153] Registering CRI-O factory Oct 30 22:01:48.862289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.862264 5226 factory.go:223] Registration of the crio container factory successfully Oct 30 22:01:48.862414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.862331 5226 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 22:01:48.862414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.862355 5226 factory.go:103] Registering Raw factory Oct 30 22:01:48.862414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.862372 5226 manager.go:1199] Started watching for new ooms in manager Oct 30 22:01:48.862789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.862772 5226 manager.go:319] Starting recovery of all containers Oct 30 22:01:48.863247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863194 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b8fc6dd-36fe-4714-bcda-1863280a8324" volumeName="kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 22:01:48.863342 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863324 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3058b5f8-ae0e-4193-b4dd-6d4abace3282" volumeName="kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 22:01:48.863421 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863366 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3058b5f8-ae0e-4193-b4dd-6d4abace3282" volumeName="kubernetes.io/empty-dir/3058b5f8-ae0e-4193-b4dd-6d4abace3282-audit-log" seLinuxMountContext="" Oct 30 22:01:48.863421 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863402 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70" volumeName="kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-certificates" seLinuxMountContext="" Oct 30 22:01:48.863421 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863417 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d531260-0f02-4b21-af0a-ec862fca540d" volumeName="kubernetes.io/secret/8d531260-0f02-4b21-af0a-ec862fca540d-cert" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863459 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dce0456-dccb-4e0f-bdbd-978371813dc4" volumeName="kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-metrics-client-ca" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863477 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863490 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863518 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089436e9-8607-4901-9b23-8e0512aacc3c" volumeName="kubernetes.io/projected/089436e9-8607-4901-9b23-8e0512aacc3c-kube-api-access-lgvlp" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863532 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3058b5f8-ae0e-4193-b4dd-6d4abace3282" volumeName="kubernetes.io/projected/3058b5f8-ae0e-4193-b4dd-6d4abace3282-kube-api-access-wpbrr" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863543 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3058b5f8-ae0e-4193-b4dd-6d4abace3282" volumeName="kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863555 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3058b5f8-ae0e-4193-b4dd-6d4abace3282" volumeName="kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863569 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba98bdc2-2e27-4499-8182-4abf75b94a23" volumeName="kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863585 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c986bec8-11a1-4c93-86fa-40b486199598" volumeName="kubernetes.io/empty-dir/c986bec8-11a1-4c93-86fa-40b486199598-tmp-dir" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863601 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfbc34e7-48ae-456a-ad03-408b46e20053" volumeName="kubernetes.io/projected/cfbc34e7-48ae-456a-ad03-408b46e20053-kube-api-access-hcv94" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863612 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863626 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f434211-8bee-4801-9c8e-39f0bb0e71d6" volumeName="kubernetes.io/projected/8f434211-8bee-4801-9c8e-39f0bb0e71d6-kube-api-access-445ts" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863638 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089436e9-8607-4901-9b23-8e0512aacc3c" volumeName="kubernetes.io/configmap/089436e9-8607-4901-9b23-8e0512aacc3c-config-volume" seLinuxMountContext="" Oct 30 22:01:48.863646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863652 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b8fc6dd-36fe-4714-bcda-1863280a8324" volumeName="kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863666 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dce0456-dccb-4e0f-bdbd-978371813dc4" volumeName="kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863787 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dce0456-dccb-4e0f-bdbd-978371813dc4" volumeName="kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-client-tls" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863820 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfbc34e7-48ae-456a-ad03-408b46e20053" volumeName="kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863832 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/configmap/10f4cba6-57dc-4397-a602-21a3bad21b82-metrics-client-ca" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863845 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20b0ee9f-f99d-472a-a1d8-df44248d9aec" volumeName="kubernetes.io/secret/20b0ee9f-f99d-472a-a1d8-df44248d9aec-agent-certs" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863859 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3058b5f8-ae0e-4193-b4dd-6d4abace3282" volumeName="kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-client-ca-bundle" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863871 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f522003-5ca0-45d8-b32e-b52db6916aca" volumeName="kubernetes.io/empty-dir/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-textfile" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863882 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8420c80f-804e-468e-8df2-fb1ec145d318" volumeName="kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-etc-tuned" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863933 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94ca8590-27e5-497d-9bb8-cb5041ceb267" volumeName="kubernetes.io/configmap/94ca8590-27e5-497d-9bb8-cb5041ceb267-iptables-alerter-script" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863951 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94ca8590-27e5-497d-9bb8-cb5041ceb267" volumeName="kubernetes.io/projected/94ca8590-27e5-497d-9bb8-cb5041ceb267-kube-api-access-g8jq7" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.863963 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dce0456-dccb-4e0f-bdbd-978371813dc4" volumeName="kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864014 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c986bec8-11a1-4c93-86fa-40b486199598" volumeName="kubernetes.io/projected/c986bec8-11a1-4c93-86fa-40b486199598-kube-api-access-vfzp9" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864030 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864057 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-grpc-tls" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864070 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f522003-5ca0-45d8-b32e-b52db6916aca" volumeName="kubernetes.io/projected/3f522003-5ca0-45d8-b32e-b52db6916aca-kube-api-access-5bhtx" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864082 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f522003-5ca0-45d8-b32e-b52db6916aca" volumeName="kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864095 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70" volumeName="kubernetes.io/empty-dir/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-ca-trust-extracted" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864109 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864122 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70" volumeName="kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-installation-pull-secrets" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864133 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8420c80f-804e-468e-8df2-fb1ec145d318" volumeName="kubernetes.io/projected/8420c80f-804e-468e-8df2-fb1ec145d318-kube-api-access-dd8k8" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864144 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba98bdc2-2e27-4499-8182-4abf75b94a23" volumeName="kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864157 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a72d78e-c521-4dba-b551-5970a7505245" volumeName="kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-config" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864171 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b8fc6dd-36fe-4714-bcda-1863280a8324" volumeName="kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-binary-copy" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864186 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3b54602-8b76-49b7-b295-42969da3c4fd" volumeName="kubernetes.io/configmap/b3b54602-8b76-49b7-b295-42969da3c4fd-serviceca" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864197 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f434211-8bee-4801-9c8e-39f0bb0e71d6" volumeName="kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cni-binary-copy" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864226 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a72d78e-c521-4dba-b551-5970a7505245" volumeName="kubernetes.io/projected/1a72d78e-c521-4dba-b551-5970a7505245-kube-api-access-5gpgg" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864238 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2675c1b0-71d1-40b8-b78c-5ef4a59389d7" volumeName="kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864251 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f522003-5ca0-45d8-b32e-b52db6916aca" volumeName="kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-metrics-client-ca" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864269 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8420c80f-804e-468e-8df2-fb1ec145d318" volumeName="kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-tmp" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864282 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3b54602-8b76-49b7-b295-42969da3c4fd" volumeName="kubernetes.io/projected/b3b54602-8b76-49b7-b295-42969da3c4fd-kube-api-access-nkwvd" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864294 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864306 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864319 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864333 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20b0ee9f-f99d-472a-a1d8-df44248d9aec" volumeName="kubernetes.io/configmap/20b0ee9f-f99d-472a-a1d8-df44248d9aec-konnectivity-ca" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864347 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70" volumeName="kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-trusted-ca" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864360 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba98bdc2-2e27-4499-8182-4abf75b94a23" volumeName="kubernetes.io/empty-dir/ba98bdc2-2e27-4499-8182-4abf75b94a23-data-volume" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864372 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864416 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config" seLinuxMountContext="" Oct 30 22:01:48.864407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864429 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089436e9-8607-4901-9b23-8e0512aacc3c" volumeName="kubernetes.io/empty-dir/089436e9-8607-4901-9b23-8e0512aacc3c-tmp-dir" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864442 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/projected/10f4cba6-57dc-4397-a602-21a3bad21b82-kube-api-access-d9bbq" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864457 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a72d78e-c521-4dba-b551-5970a7505245" volumeName="kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-script-lib" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864469 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f522003-5ca0-45d8-b32e-b52db6916aca" volumeName="kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864482 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="588b62a8-9745-4e5a-97dd-f8c451197018" volumeName="kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864494 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70" volumeName="kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-bound-sa-token" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864507 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dce0456-dccb-4e0f-bdbd-978371813dc4" volumeName="kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864521 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864534 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864550 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864563 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f522003-5ca0-45d8-b32e-b52db6916aca" volumeName="kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-tls" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864583 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70" volumeName="kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-image-registry-private-configuration" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864598 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864625 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a72d78e-c521-4dba-b551-5970a7505245" volumeName="kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-env-overrides" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864638 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3058b5f8-ae0e-4193-b4dd-6d4abace3282" volumeName="kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864654 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d531260-0f02-4b21-af0a-ec862fca540d" volumeName="kubernetes.io/projected/8d531260-0f02-4b21-af0a-ec862fca540d-kube-api-access-wsdpq" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864666 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dce0456-dccb-4e0f-bdbd-978371813dc4" volumeName="kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864679 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="089436e9-8607-4901-9b23-8e0512aacc3c" volumeName="kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864692 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f4cba6-57dc-4397-a602-21a3bad21b82" volumeName="kubernetes.io/empty-dir/10f4cba6-57dc-4397-a602-21a3bad21b82-tmp" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864704 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a72d78e-c521-4dba-b551-5970a7505245" volumeName="kubernetes.io/secret/1a72d78e-c521-4dba-b551-5970a7505245-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864716 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70" volumeName="kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-tls" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864732 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dce0456-dccb-4e0f-bdbd-978371813dc4" volumeName="kubernetes.io/projected/9dce0456-dccb-4e0f-bdbd-978371813dc4-kube-api-access-l4ccz" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864745 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba98bdc2-2e27-4499-8182-4abf75b94a23" volumeName="kubernetes.io/projected/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-api-access-kppgx" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864757 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70" volumeName="kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-kube-api-access-2l64c" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864768 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adebb4b-a9a3-4c8e-89c2-940cfcab9e28" volumeName="kubernetes.io/projected/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kube-api-access-8kxtv" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864781 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f434211-8bee-4801-9c8e-39f0bb0e71d6" volumeName="kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-daemon-config" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864818 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864833 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864850 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864865 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd5821b-7635-4bbf-9e07-86fa1579328d" volumeName="kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864878 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b8fc6dd-36fe-4714-bcda-1863280a8324" volumeName="kubernetes.io/projected/1b8fc6dd-36fe-4714-bcda-1863280a8324-kube-api-access-6frbn" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864892 5226 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dce0456-dccb-4e0f-bdbd-978371813dc4" volumeName="kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-federate-client-tls" seLinuxMountContext="" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864904 5226 reconstruct.go:97] "Volume reconstruction finished" Oct 30 22:01:48.865466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.864912 5226 reconciler.go:29] "Reconciler: start to sync state" Oct 30 22:01:48.895363 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.895302 5226 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 22:01:48.896910 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.896882 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:01:48.897096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.897075 5226 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 22:01:48.897166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.897100 5226 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 22:01:48.897166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.897123 5226 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 22:01:48.897266 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:48.897163 5226 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 22:01:48.897920 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.897901 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:01:48.918554 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.918488 5226 generic.go:356] "Generic (PLEG): container finished" podID="56d00c89f3615755885c52b7effe8bce" containerID="7db276559323dec53ac72505a6e0b35e52c9461b2a27e548f3227fa3c2f56245" exitCode=0 Oct 30 22:01:48.924716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.924679 5226 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="7d57a31e142d1602e5b2ce3936519b1a128beccd91cfc6a901ee970bb4f32c51" exitCode=0 Oct 30 22:01:48.924716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.924712 5226 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="a964019c4d554a73913785604c10a17851d12dbbe1bb15abb107a99303099ee3" exitCode=0 Oct 30 22:01:48.924923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.924723 5226 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="ad94505a67e20355a15dcec7434754fb937ba9e23343b15b8624c86e42fdd9e7" exitCode=0 Oct 30 22:01:48.924923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.924734 5226 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="c1bb2944688e30dee0178c7ca7f02d04b70dd01a04f808ec3e9ed528dd63344d" exitCode=0 Oct 30 22:01:48.924923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.924743 5226 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="ea2a3b696913e2bae88aee6ed511f3ed09b23f95c9d170682ad46aca953e4ac3" exitCode=0 Oct 30 22:01:48.924923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.924751 5226 generic.go:356] "Generic (PLEG): container finished" podID="1b8fc6dd-36fe-4714-bcda-1863280a8324" containerID="479e3ffb4412d2c5114e9280ddcb7fc0c7fa9e61ef8d5aa82beb180dfe7bf76f" exitCode=0 Oct 30 22:01:48.931705 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.931607 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:01:48.932003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.931973 5226 generic.go:356] "Generic (PLEG): container finished" podID="1a72d78e-c521-4dba-b551-5970a7505245" containerID="d10cb107b004efd16c861b75966a655db82f854206f6d5f81b8c2e26d4177553" exitCode=1 Oct 30 22:01:48.935057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.935031 5226 generic.go:356] "Generic (PLEG): container finished" podID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerID="8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0" exitCode=0 Oct 30 22:01:48.937194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.937171 5226 generic.go:356] "Generic (PLEG): container finished" podID="3f522003-5ca0-45d8-b32e-b52db6916aca" containerID="c180196eeec6d25c1315c2abc16b5a4c02c5c0b7df639e2d2cba113e163b31c9" exitCode=0 Oct 30 22:01:48.958141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.958118 5226 manager.go:324] Recovery completed Oct 30 22:01:48.958903 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:48.958885 5226 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" Oct 30 22:01:48.976471 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.976445 5226 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:48.978612 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.978592 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:48.978713 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.978623 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:48.978713 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.978634 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:48.981562 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981540 5226 cpu_manager.go:226] "Starting" policy="none" Oct 30 22:01:48.981562 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981554 5226 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:48.981702 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981557 5226 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 22:01:48.981702 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981616 5226 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 22:01:48.981810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981735 5226 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 22:01:48.981810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981747 5226 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 22:01:48.981810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981765 5226 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 22:01:48.981810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981771 5226 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 22:01:48.981810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981777 5226 policy_none.go:50] "Start" Oct 30 22:01:48.981810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981789 5226 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 22:01:48.981810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.981798 5226 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 22:01:48.982656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.982604 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:48.982656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.982637 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:48.982656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.982652 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:48.986088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.986072 5226 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 22:01:48.986088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.986090 5226 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 22:01:48.986197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.986097 5226 policy_none.go:44] "Start" Oct 30 22:01:48.991013 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.990986 5226 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 22:01:48.993831 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.993817 5226 manager.go:341] "Starting Device Plugin manager" Oct 30 22:01:48.993910 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:48.993851 5226 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 22:01:48.993910 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.993863 5226 server.go:87] "Starting device plugin registration server" Oct 30 22:01:48.994167 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.994153 5226 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 22:01:48.994242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.994170 5226 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 22:01:48.994307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.994283 5226 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 22:01:48.994450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.994411 5226 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 22:01:48.994572 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.994463 5226 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 22:01:48.994572 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.994470 5226 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 22:01:48.995077 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.994504 5226 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:01:48.994444667Z","UUID":"99df35b9-a62a-41b0-9cc6-162be81b319e","Handler":null,"Name":"","Endpoint":""} Oct 30 22:01:48.995993 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:48.995971 5226 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 22:01:48.996074 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:48.996009 5226 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" Oct 30 22:01:48.996482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.996341 5226 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 22:01:48.996482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.996366 5226 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 22:01:48.997313 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.997283 5226 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal"] Oct 30 22:01:48.997410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.997395 5226 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:48.998489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.998451 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:48.998489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.998483 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:48.998650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.998500 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:48.998972 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.998948 5226 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:48.999084 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.999065 5226 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:49.000012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.999994 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000020 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:48.999994 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000036 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000047 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000061 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:49.000240 5226 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000378 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"7419d42fafb37e8b2d87d41a6e268c81","Type":"ContainerStarted","Data":"4374886a35c8323924d218089199e7dd03e588a58b73fe136de505d0fa2c596b"} Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000411 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"7419d42fafb37e8b2d87d41a6e268c81","Type":"ContainerStarted","Data":"4642ae4eba01be1299c42eb440d862b9af8b1a2089e6d7b0f1ada23dedd6c357"} Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000423 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"56d00c89f3615755885c52b7effe8bce","Type":"ContainerStarted","Data":"0fbb2c1a4d375e6afe4b60873476df8ba0a6d3cf5a2eae6aaa3eea1820012837"} Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000436 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"56d00c89f3615755885c52b7effe8bce","Type":"ContainerDied","Data":"7db276559323dec53ac72505a6e0b35e52c9461b2a27e548f3227fa3c2f56245"} Oct 30 22:01:49.000453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000450 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" event={"ID":"56d00c89f3615755885c52b7effe8bce","Type":"ContainerStarted","Data":"0cfbc5e5215ff32bb6d5854f5dcd92f3efc54b9be9172cde3ab1344e2176025c"} Oct 30 22:01:49.000847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.000526 5226 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:49.001435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.001413 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:49.001511 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.001439 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:49.001511 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.001449 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:49.001635 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:49.001574 5226 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-194.us-west-2.compute.internal\" not found" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.029385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.029357 5226 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:01:49.029513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.029400 5226 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:01:49.029513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.029411 5226 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 22:01:49.030161 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.030146 5226 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:01:49.059309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.059245 5226 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.094891 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.094858 5226 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 22:01:49.096102 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.096081 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 22:01:49.096258 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.096118 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 22:01:49.096258 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.096134 5226 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-194.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 22:01:49.096258 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.096198 5226 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.121751 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.121711 5226 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 22:01:49.130196 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.130158 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.130196 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.130196 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.130372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.130229 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7419d42fafb37e8b2d87d41a6e268c81-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"7419d42fafb37e8b2d87d41a6e268c81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.135951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.135927 5226 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:01:49.136013 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:49.135986 5226 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.136013 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.136001 5226 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.210097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.210069 5226 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.210285 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.210161 5226 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.221861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.221827 5226 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:01:49.222016 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:49.221883 5226 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.230997 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.230969 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.231044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.231004 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.231044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.231028 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7419d42fafb37e8b2d87d41a6e268c81-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"7419d42fafb37e8b2d87d41a6e268c81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.231111 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.231079 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7419d42fafb37e8b2d87d41a6e268c81-config\") pod \"kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"7419d42fafb37e8b2d87d41a6e268c81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.231111 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.231080 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.231169 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.231080 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d00c89f3615755885c52b7effe8bce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" (UID: \"56d00c89f3615755885c52b7effe8bce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.831371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.831334 5226 apiserver.go:52] "Watching apiserver" Oct 30 22:01:49.935725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.935691 5226 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 22:01:49.936370 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.936345 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-cbb75f5c9-k9gj6","openshift-network-diagnostics/network-check-target-6mdvj","openshift-monitoring/node-exporter-h7dlr","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-additional-cni-plugins-trblz","openshift-multus/multus-xc24q","openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d","openshift-dns/node-resolver-rg6kq","openshift-ingress-canary/ingress-canary-l4pxs","kube-system/konnectivity-agent-4k4pr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2","openshift-cluster-node-tuning-operator/tuned-c44r7","openshift-insights/insights-runtime-extractor-5w4t9","openshift-monitoring/thanos-querier-665bfc67b6-srdl5","openshift-image-registry/image-registry-949b9f975-65pwb","kube-system/global-pull-secret-syncer-h8pdj","kube-system/kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal","openshift-dns/dns-default-l6v6f","openshift-image-registry/node-ca-47nqg","openshift-network-operator/iptables-alerter-2qtcg","openshift-ovn-kubernetes/ovnkube-node-mlgss","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-5qc57"] Oct 30 22:01:49.947411 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.947381 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:49.947920 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.947900 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:49.956091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.956067 5226 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:49.961139 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.961111 5226 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 22:01:49.992698 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.992657 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 22:01:49.992891 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.992781 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 22:01:49.993034 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993015 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:01:49.993238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993216 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 22:01:49.993309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993231 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 22:01:49.993309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993257 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 22:01:49.993309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993286 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 22:01:49.993309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993293 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 22:01:49.993495 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993372 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 22:01:49.993495 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993445 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 22:01:49.993643 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993603 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 22:01:49.993772 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993759 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 22:01:49.993969 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.993950 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 22:01:49.994064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.994047 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 22:01:49.994112 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.994054 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 22:01:49.994605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.994592 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 22:01:49.994774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.994762 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 22:01:49.995230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.995218 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 22:01:49.995417 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.995406 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 22:01:49.995555 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.995541 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 22:01:49.995679 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.995668 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 22:01:49.995771 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.995760 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 22:01:49.995876 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.995844 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 22:01:49.995929 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.995918 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 22:01:49.996400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.996388 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 22:01:49.998308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:49.998293 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:01:50.031082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.031052 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 22:01:50.031601 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.031587 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 22:01:50.031763 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.031750 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 22:01:50.033133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.033116 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 22:01:50.035449 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035431 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-sys\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.035490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035461 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsdpq\" (UniqueName: \"kubernetes.io/projected/8d531260-0f02-4b21-af0a-ec862fca540d-kube-api-access-wsdpq\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:50.035490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035483 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-var-lib-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.035566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035501 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.035607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035560 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.035759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035733 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9bbq\" (UniqueName: \"kubernetes.io/projected/10f4cba6-57dc-4397-a602-21a3bad21b82-kube-api-access-d9bbq\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.035817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035784 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.035877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035813 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-bin\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.035877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035870 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-modprobe-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.035952 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035908 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.035986 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.035969 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.036072 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036051 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-grpc-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.036125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036112 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.036179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036147 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:50.036179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036120 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 22:01:50.036289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036256 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:50.036289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036281 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.036381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036304 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-etc-tuned\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.036381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036326 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089436e9-8607-4901-9b23-8e0512aacc3c-tmp-dir\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:50.036381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036344 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbrr\" (UniqueName: \"kubernetes.io/projected/3058b5f8-ae0e-4193-b4dd-6d4abace3282-kube-api-access-wpbrr\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.036381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036366 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.036571 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036392 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-multus-certs\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.036571 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036447 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ca8590-27e5-497d-9bb8-cb5041ceb267-host-slash\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:01:50.036571 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036465 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089436e9-8607-4901-9b23-8e0512aacc3c-tmp-dir\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:50.036571 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036565 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.036762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036666 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20b0ee9f-f99d-472a-a1d8-df44248d9aec-konnectivity-ca\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:01:50.036762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036721 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.036856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036772 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-var-lib-kubelet\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.036856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036820 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-etc-tuned\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.036856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036805 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kxtv\" (UniqueName: \"kubernetes.io/projected/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kube-api-access-8kxtv\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.037012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036859 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-textfile\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.037012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036950 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.037012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.036963 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-textfile\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.037148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037059 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-cnibin\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.037148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037104 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-tls\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.037148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037132 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c986bec8-11a1-4c93-86fa-40b486199598-tmp-dir\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:01:50.037289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037171 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.037289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037196 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:50.037289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037225 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c986bec8-11a1-4c93-86fa-40b486199598-tmp-dir\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:01:50.037289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037250 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-sys\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.037429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037345 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20b0ee9f-f99d-472a-a1d8-df44248d9aec-konnectivity-ca\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:01:50.037429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037365 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.037429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037413 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-kubelet\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.037548 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037453 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.037548 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037491 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a72d78e-c521-4dba-b551-5970a7505245-ovn-node-metrics-cert\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.037645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037589 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-host\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.037645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037617 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-daemon-config\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.037727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037661 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445ts\" (UniqueName: \"kubernetes.io/projected/8f434211-8bee-4801-9c8e-39f0bb0e71d6-kube-api-access-445ts\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.037727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037676 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-root\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.037727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037696 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20b0ee9f-f99d-472a-a1d8-df44248d9aec-agent-certs\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:01:50.037727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037721 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysconfig\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.037910 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037743 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.037910 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037765 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-conf\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.037910 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037828 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.037910 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037896 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcv94\" (UniqueName: \"kubernetes.io/projected/cfbc34e7-48ae-456a-ad03-408b46e20053-kube-api-access-hcv94\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:50.037910 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037906 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-daemon-config\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.038109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037937 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-systemd-units\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.038109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037962 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-bound-sa-token\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.038109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.037986 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.038109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038022 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f4cba6-57dc-4397-a602-21a3bad21b82-tmp\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.038109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038030 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.038289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038106 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 22:01:50.038289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038107 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f4cba6-57dc-4397-a602-21a3bad21b82-tmp\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.038289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038128 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b54602-8b76-49b7-b295-42969da3c4fd-host\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:01:50.038493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038153 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-systemd\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.038594 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038574 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.038670 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038648 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.038717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038676 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.038773 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038755 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-kubelet-config\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:50.038818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038799 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba98bdc2-2e27-4499-8182-4abf75b94a23-crio-socket\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.038861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038831 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-run\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.038911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038866 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/089436e9-8607-4901-9b23-8e0512aacc3c-config-volume\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:50.038995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.038977 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.039051 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039012 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-ca-trust-extracted\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.039117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039081 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-sys-fs\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.039183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039121 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.039183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039152 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-netns\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.039322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039185 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-wtmp\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.039322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039288 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-trusted-ca\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.039414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039328 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.039414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039361 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.039414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039401 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-netns\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.039544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039436 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-kubernetes\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.039544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039468 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-client-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.039544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039495 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jq7\" (UniqueName: \"kubernetes.io/projected/94ca8590-27e5-497d-9bb8-cb5041ceb267-kube-api-access-g8jq7\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:01:50.039677 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039622 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzp9\" (UniqueName: \"kubernetes.io/projected/c986bec8-11a1-4c93-86fa-40b486199598-kube-api-access-vfzp9\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:01:50.039677 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039656 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6frbn\" (UniqueName: \"kubernetes.io/projected/1b8fc6dd-36fe-4714-bcda-1863280a8324-kube-api-access-6frbn\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.039764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039683 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-serving-certs-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.039764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039715 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cni-binary-copy\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.039764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039748 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-conf-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.039887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039781 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/94ca8590-27e5-497d-9bb8-cb5041ceb267-iptables-alerter-script\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:01:50.039887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039810 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-slash\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.039887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039833 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.039887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039866 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-etc-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.040048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039904 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-node-log\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.040048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039933 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-log-socket\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.040048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.039967 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-certificates\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.040048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040001 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.040048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040032 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d531260-0f02-4b21-af0a-ec862fca540d-cert\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:50.040307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040288 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.040356 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040328 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-ovn\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.040403 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040360 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.040403 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040397 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.040492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040427 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.040492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040454 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-bin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.040576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040494 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-tls\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.040576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040530 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:50.040576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040562 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-kubelet\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.040704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040604 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-script-lib\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.040704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040631 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-registration-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.040704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040667 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gprrl\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.040704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040698 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhtx\" (UniqueName: \"kubernetes.io/projected/3f522003-5ca0-45d8-b32e-b52db6916aca-kube-api-access-5bhtx\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.040871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040729 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwvd\" (UniqueName: \"kubernetes.io/projected/b3b54602-8b76-49b7-b295-42969da3c4fd-kube-api-access-nkwvd\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:01:50.040871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040729 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 22:01:50.040871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040755 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-lib-modules\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.040871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040790 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-image-registry-private-configuration\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.040871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040823 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.040871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040865 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-system-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.041117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040893 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppgx\" (UniqueName: \"kubernetes.io/projected/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-api-access-kppgx\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.041117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040926 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-config\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.041117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040960 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpgg\" (UniqueName: \"kubernetes.io/projected/1a72d78e-c521-4dba-b551-5970a7505245-kube-api-access-5gpgg\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.041117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040992 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-tmp\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.041117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.040992 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/94ca8590-27e5-497d-9bb8-cb5041ceb267-iptables-alerter-script\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:01:50.041117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041016 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 22:01:50.041117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041056 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cnibin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.041117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041101 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-socket-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041146 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvlp\" (UniqueName: \"kubernetes.io/projected/089436e9-8607-4901-9b23-8e0512aacc3c-kube-api-access-lgvlp\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041184 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041195 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041234 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-installation-pull-secrets\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041276 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-client-certs\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041295 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-certificates\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041311 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3058b5f8-ae0e-4193-b4dd-6d4abace3282-audit-log\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041352 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ccz\" (UniqueName: \"kubernetes.io/projected/9dce0456-dccb-4e0f-bdbd-978371813dc4-kube-api-access-l4ccz\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041383 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041420 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041421 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-etc-kubernetes\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.041515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041459 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c986bec8-11a1-4c93-86fa-40b486199598-hosts-file\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:01:50.041995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041386 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 22:01:50.041995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041742 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 22:01:50.041995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041847 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3058b5f8-ae0e-4193-b4dd-6d4abace3282-audit-log\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.041995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041923 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-script-lib\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.041995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.041966 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-systemd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.042220 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042004 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-socket-dir-parent\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.042220 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042031 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-dbus\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:50.042220 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042065 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-netd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.042220 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042112 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.042220 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042183 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042198 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-ca-trust-extracted\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042234 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-system-cni-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042270 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-tls\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042307 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-federate-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042350 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042357 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8420c80f-804e-468e-8df2-fb1ec145d318-tmp\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042387 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-hostroot\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042408 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/089436e9-8607-4901-9b23-8e0512aacc3c-config-volume\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:50.042435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042428 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-metrics-client-ca\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.042860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042838 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-serving-certs-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.042912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.042859 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.043046 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043028 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cni-binary-copy\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.043126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043077 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-metrics-client-ca\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.043501 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043476 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.043742 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043603 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.043992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043747 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.043992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043776 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-multus\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.043992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043799 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3b54602-8b76-49b7-b295-42969da3c4fd-serviceca\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:01:50.043992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043928 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba98bdc2-2e27-4499-8182-4abf75b94a23-data-volume\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.044923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.043994 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-binary-copy\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.045025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.044368 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-ovnkube-config\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.045025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.044372 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba98bdc2-2e27-4499-8182-4abf75b94a23-data-volume\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.045025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.044687 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b8fc6dd-36fe-4714-bcda-1863280a8324-cni-binary-copy\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.045025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.044970 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3b54602-8b76-49b7-b295-42969da3c4fd-serviceca\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:01:50.045025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.044977 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-env-overrides\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.045236 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045065 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8k8\" (UniqueName: \"kubernetes.io/projected/8420c80f-804e-468e-8df2-fb1ec145d318-kube-api-access-dd8k8\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.045466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045096 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l64c\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-kube-api-access-2l64c\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.045518 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045499 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10f4cba6-57dc-4397-a602-21a3bad21b82-metrics-client-ca\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.045562 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045525 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 22:01:50.045562 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045531 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a72d78e-c521-4dba-b551-5970a7505245-env-overrides\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.045562 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045532 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-metrics-server-audit-profiles\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.045696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045587 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-device-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.045696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045634 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.045696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045671 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-k8s-cni-cncf-io\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.045824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045768 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10f4cba6-57dc-4397-a602-21a3bad21b82-metrics-client-ca\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.045824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045774 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 22:01:50.045912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.045846 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3058b5f8-ae0e-4193-b4dd-6d4abace3282-metrics-server-audit-profiles\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.047133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.047109 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.048219 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.048178 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.049320 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.049304 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.053682 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.053662 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f522003-5ca0-45d8-b32e-b52db6916aca-metrics-client-ca\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.054734 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.054716 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-metrics-client-ca\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.083325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.083194 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 22:01:50.083558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.083499 5226 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 22:01:50.083897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.083855 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 22:01:50.083897 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:50.083817 5226 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-194.us-west-2.compute.internal" Oct 30 22:01:50.084161 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084139 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 22:01:50.084295 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084277 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 22:01:50.084360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084306 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xd6xx\"" Oct 30 22:01:50.084360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084306 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 22:01:50.084360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084339 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 22:01:50.084360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084348 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084377 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084399 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6qlsl\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084407 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-75hcr\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084425 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skckn\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084427 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6j7ll\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084459 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084455 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084500 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-122dschqev18o\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084518 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084536 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pd25c\"" Oct 30 22:01:50.084558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084547 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084602 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084608 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084648 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084651 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084694 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084700 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084712 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084716 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1pbn1s61uifeu\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084736 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084738 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084758 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wf9b4\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084805 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qp8bv\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084825 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dj92j\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084874 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084914 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hlw52\"" Oct 30 22:01:50.084956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.084960 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qdfn5\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085007 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085010 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085060 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085130 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfj4m\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085188 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rstdk\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085239 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-skl9p\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085283 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rw2ld\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085368 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gzw7r\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085448 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085455 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 22:01:50.085522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085489 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sqg6l\"" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085546 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085614 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085685 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v5kds\"" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085756 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085825 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085860 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085902 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085944 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.085978 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 22:01:50.086010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086012 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 22:01:50.086304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086026 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.086304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086040 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hpnbl\"" Oct 30 22:01:50.086304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086086 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xdcch\"" Oct 30 22:01:50.086304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086139 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g5t42\"" Oct 30 22:01:50.086304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086224 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-telemeter-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.086304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086265 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.086470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086409 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.086546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086532 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-grpc-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.086546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.086539 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-original-pull-secret\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:50.087600 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.087581 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-tls\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.087672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.087660 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/089436e9-8607-4901-9b23-8e0512aacc3c-metrics-tls\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:50.087724 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.087703 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.087781 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.087768 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a72d78e-c521-4dba-b551-5970a7505245-ovn-node-metrics-cert\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.088841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.088827 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20b0ee9f-f99d-472a-a1d8-df44248d9aec-agent-certs\") pod \"konnectivity-agent-4k4pr\" (UID: \"20b0ee9f-f99d-472a-a1d8-df44248d9aec\") " pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:01:50.091377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.091356 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.092546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092526 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfbc34e7-48ae-456a-ad03-408b46e20053-metrics-certs\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:50.092618 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092535 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-tls\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.092650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092637 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d531260-0f02-4b21-af0a-ec862fca540d-cert\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:50.092680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092637 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-secret-metrics-server-client-certs\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.092709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092640 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.092742 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092733 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-secret-telemeter-client\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.092772 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092755 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.092810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092765 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.092872 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.092859 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3058b5f8-ae0e-4193-b4dd-6d4abace3282-client-ca-bundle\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.093736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093701 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba98bdc2-2e27-4499-8182-4abf75b94a23-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.093736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093721 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9dce0456-dccb-4e0f-bdbd-978371813dc4-federate-client-tls\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.093885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093807 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.093885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093810 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-registry-tls\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.093885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093837 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.093885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093854 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-image-registry-private-configuration\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.093885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093866 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.094115 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093943 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.094115 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093949 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-installation-pull-secrets\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.094115 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.093961 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-trusted-ca\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.094115 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.094023 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.094115 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.094077 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/10f4cba6-57dc-4397-a602-21a3bad21b82-secret-thanos-querier-tls\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.094918 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.094901 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9bbq\" (UniqueName: \"kubernetes.io/projected/10f4cba6-57dc-4397-a602-21a3bad21b82-kube-api-access-d9bbq\") pod \"thanos-querier-665bfc67b6-srdl5\" (UID: \"10f4cba6-57dc-4397-a602-21a3bad21b82\") " pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:50.095193 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.095170 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsdpq\" (UniqueName: \"kubernetes.io/projected/8d531260-0f02-4b21-af0a-ec862fca540d-kube-api-access-wsdpq\") pod \"ingress-canary-l4pxs\" (UID: \"8d531260-0f02-4b21-af0a-ec862fca540d\") " pod="openshift-ingress-canary/ingress-canary-l4pxs" Oct 30 22:01:50.095245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.095193 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgj8\" (UniqueName: \"kubernetes.io/projected/588b62a8-9745-4e5a-97dd-f8c451197018-kube-api-access-gzgj8\") pod \"network-check-target-6mdvj\" (UID: \"588b62a8-9745-4e5a-97dd-f8c451197018\") " pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:50.095639 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.095620 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcv94\" (UniqueName: \"kubernetes.io/projected/cfbc34e7-48ae-456a-ad03-408b46e20053-kube-api-access-hcv94\") pod \"network-metrics-daemon-5qc57\" (UID: \"cfbc34e7-48ae-456a-ad03-408b46e20053\") " pod="openshift-multus/network-metrics-daemon-5qc57" Oct 30 22:01:50.095972 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.095956 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-bound-sa-token\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.096135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.096120 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445ts\" (UniqueName: \"kubernetes.io/projected/8f434211-8bee-4801-9c8e-39f0bb0e71d6-kube-api-access-445ts\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.100445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.100421 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jq7\" (UniqueName: \"kubernetes.io/projected/94ca8590-27e5-497d-9bb8-cb5041ceb267-kube-api-access-g8jq7\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:01:50.100534 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.100516 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6frbn\" (UniqueName: \"kubernetes.io/projected/1b8fc6dd-36fe-4714-bcda-1863280a8324-kube-api-access-6frbn\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.100576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.100551 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gprrl\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl\") pod \"alertmanager-main-0\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:01:50.100965 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.100946 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppgx\" (UniqueName: \"kubernetes.io/projected/ba98bdc2-2e27-4499-8182-4abf75b94a23-kube-api-access-kppgx\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.101508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.101488 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpgg\" (UniqueName: \"kubernetes.io/projected/1a72d78e-c521-4dba-b551-5970a7505245-kube-api-access-5gpgg\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.101600 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.101580 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhtx\" (UniqueName: \"kubernetes.io/projected/3f522003-5ca0-45d8-b32e-b52db6916aca-kube-api-access-5bhtx\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.101668 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.101651 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ccz\" (UniqueName: \"kubernetes.io/projected/9dce0456-dccb-4e0f-bdbd-978371813dc4-kube-api-access-l4ccz\") pod \"telemeter-client-7fc79ffd54-xnz4d\" (UID: \"9dce0456-dccb-4e0f-bdbd-978371813dc4\") " pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.101712 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.101672 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwvd\" (UniqueName: \"kubernetes.io/projected/b3b54602-8b76-49b7-b295-42969da3c4fd-kube-api-access-nkwvd\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:01:50.104769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.104752 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l64c\" (UniqueName: \"kubernetes.io/projected/6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70-kube-api-access-2l64c\") pod \"image-registry-949b9f975-65pwb\" (UID: \"6ef08e87-e62d-4ff9-9b6c-e90c7bf48b70\") " pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:50.105324 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.105303 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzp9\" (UniqueName: \"kubernetes.io/projected/c986bec8-11a1-4c93-86fa-40b486199598-kube-api-access-vfzp9\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:01:50.105324 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.105314 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kxtv\" (UniqueName: \"kubernetes.io/projected/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kube-api-access-8kxtv\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.105420 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.105405 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbrr\" (UniqueName: \"kubernetes.io/projected/3058b5f8-ae0e-4193-b4dd-6d4abace3282-kube-api-access-wpbrr\") pod \"metrics-server-cbb75f5c9-k9gj6\" (UID: \"3058b5f8-ae0e-4193-b4dd-6d4abace3282\") " pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.105490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.105470 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8k8\" (UniqueName: \"kubernetes.io/projected/8420c80f-804e-468e-8df2-fb1ec145d318-kube-api-access-dd8k8\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.105889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.105873 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvlp\" (UniqueName: \"kubernetes.io/projected/089436e9-8607-4901-9b23-8e0512aacc3c-kube-api-access-lgvlp\") pod \"dns-default-l6v6f\" (UID: \"089436e9-8607-4901-9b23-8e0512aacc3c\") " pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:50.146789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146750 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-root\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.146789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146790 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysconfig\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146806 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-conf\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146869 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysconfig\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146867 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-root\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146898 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-systemd-units\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146917 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b54602-8b76-49b7-b295-42969da3c4fd-host\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146933 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-systemd\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146951 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-kubelet-config\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146967 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba98bdc2-2e27-4499-8182-4abf75b94a23-crio-socket\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.146983 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-run\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147009 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b54602-8b76-49b7-b295-42969da3c4fd-host\") pod \"node-ca-47nqg\" (UID: \"b3b54602-8b76-49b7-b295-42969da3c4fd\") " pod="openshift-image-registry/node-ca-47nqg" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147025 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-systemd-units\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147047 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-sys-fs\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147061 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-systemd\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147034 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba98bdc2-2e27-4499-8182-4abf75b94a23-crio-socket\") pod \"insights-runtime-extractor-5w4t9\" (UID: \"ba98bdc2-2e27-4499-8182-4abf75b94a23\") " pod="openshift-insights/insights-runtime-extractor-5w4t9" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147085 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-netns\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147087 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-run\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147046 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-kubelet-config\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147017 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-conf\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147107 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-wtmp\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147127 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-sys-fs\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147139 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-netns\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147141 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-netns\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147158 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-node-exporter-wtmp\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147179 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-kubernetes\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147172 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-netns\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147236 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-conf-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147261 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-slash\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147269 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-kubernetes\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147267 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-conf-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147291 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147321 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-etc-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147343 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-node-log\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147372 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-slash\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147372 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-etc-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147378 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147386 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-log-socket\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147403 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-node-log\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147416 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147425 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-log-socket\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147432 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-ovn\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147454 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147485 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147508 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147514 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.147508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147526 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147543 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-bin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147510 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-ovn\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147563 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-kubelet\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147584 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-registration-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147601 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-lib-modules\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147605 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147613 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-bin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147621 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-system-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147646 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cnibin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147678 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-registration-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147692 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-system-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147700 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-lib-modules\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147681 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-socket-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147645 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-kubelet\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147693 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-cnibin\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147616 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-sysctl-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147720 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-socket-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147729 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-etc-kubernetes\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147749 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-etc-kubernetes\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147747 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c986bec8-11a1-4c93-86fa-40b486199598-hosts-file\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147768 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-systemd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147775 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c986bec8-11a1-4c93-86fa-40b486199598-hosts-file\") pod \"node-resolver-rg6kq\" (UID: \"c986bec8-11a1-4c93-86fa-40b486199598\") " pod="openshift-dns/node-resolver-rg6kq" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147783 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-socket-dir-parent\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147799 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-dbus\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147803 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-systemd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147816 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-netd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147861 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-system-cni-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147861 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2675c1b0-71d1-40b8-b78c-5ef4a59389d7-dbus\") pod \"global-pull-secret-syncer-h8pdj\" (UID: \"2675c1b0-71d1-40b8-b78c-5ef4a59389d7\") " pod="kube-system/global-pull-secret-syncer-h8pdj" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147860 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-socket-dir-parent\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147878 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-hostroot\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147896 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147901 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-netd\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147904 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-system-cni-dir\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147911 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-multus\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147940 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-hostroot\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147946 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-device-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147954 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-multus-cni-dir\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147971 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-k8s-cni-cncf-io\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147975 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-cni-multus\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147985 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7adebb4b-a9a3-4c8e-89c2-940cfcab9e28-device-dir\") pod \"aws-ebs-csi-driver-node-gpnm2\" (UID: \"7adebb4b-a9a3-4c8e-89c2-940cfcab9e28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpnm2" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.147996 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-sys\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148012 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-k8s-cni-cncf-io\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148021 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-var-lib-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148044 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148059 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-bin\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148073 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-modprobe-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148079 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-var-lib-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148096 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-run-openvswitch\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148102 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-multus-certs\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148115 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f522003-5ca0-45d8-b32e-b52db6916aca-sys\") pod \"node-exporter-h7dlr\" (UID: \"3f522003-5ca0-45d8-b32e-b52db6916aca\") " pod="openshift-monitoring/node-exporter-h7dlr" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148115 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-cni-bin\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148118 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ca8590-27e5-497d-9bb8-cb5041ceb267-host-slash\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148137 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ca8590-27e5-497d-9bb8-cb5041ceb267-host-slash\") pod \"iptables-alerter-2qtcg\" (UID: \"94ca8590-27e5-497d-9bb8-cb5041ceb267\") " pod="openshift-network-operator/iptables-alerter-2qtcg" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148146 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-run-multus-certs\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148172 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-var-lib-kubelet\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148194 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-cnibin\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148199 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-etc-modprobe-d\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148232 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148241 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b8fc6dd-36fe-4714-bcda-1863280a8324-cnibin\") pod \"multus-additional-cni-plugins-trblz\" (UID: \"1b8fc6dd-36fe-4714-bcda-1863280a8324\") " pod="openshift-multus/multus-additional-cni-plugins-trblz" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148257 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-var-lib-kubelet\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148261 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-sys\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148290 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-kubelet\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148301 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-sys\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148316 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-host\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148327 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f434211-8bee-4801-9c8e-39f0bb0e71d6-host-var-lib-kubelet\") pod \"multus-xc24q\" (UID: \"8f434211-8bee-4801-9c8e-39f0bb0e71d6\") " pod="openshift-multus/multus-xc24q" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148328 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a72d78e-c521-4dba-b551-5970a7505245-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlgss\" (UID: \"1a72d78e-c521-4dba-b551-5970a7505245\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.148591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.148392 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8420c80f-804e-468e-8df2-fb1ec145d318-host\") pod \"tuned-c44r7\" (UID: \"8420c80f-804e-468e-8df2-fb1ec145d318\") " pod="openshift-cluster-node-tuning-operator/tuned-c44r7" Oct 30 22:01:50.248141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.248116 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:01:50.248257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.248127 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" Oct 30 22:01:50.248371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.248353 5226 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:01:50.468020 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.467994 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:50.671808 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.671747 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d"] Oct 30 22:01:50.684342 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:50.684313 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dce0456_dccb_4e0f_bdbd_978371813dc4.slice/crio-a4fc630d6cf602daf15d78b3b76048524356173ae4bdf7dc3727d02fb2cbbda3 WatchSource:0}: Error finding container a4fc630d6cf602daf15d78b3b76048524356173ae4bdf7dc3727d02fb2cbbda3: Status 404 returned error can't find the container with id a4fc630d6cf602daf15d78b3b76048524356173ae4bdf7dc3727d02fb2cbbda3 Oct 30 22:01:50.960126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.960090 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" event={"ID":"9dce0456-dccb-4e0f-bdbd-978371813dc4","Type":"ContainerStarted","Data":"a4fc630d6cf602daf15d78b3b76048524356173ae4bdf7dc3727d02fb2cbbda3"} Oct 30 22:01:50.961390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:50.961361 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" event={"ID":"3058b5f8-ae0e-4193-b4dd-6d4abace3282","Type":"ContainerStarted","Data":"eb36f3a37ce2fb052329620d5a8990bcab49c8a6d8a76e7f7db8bfd778180a3d"} Oct 30 22:01:51.650453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.650372 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:01:51.653471 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.653445 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.704154 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.704120 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 22:01:51.704154 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.704150 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 22:01:51.704154 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.704131 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 22:01:51.709340 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.709303 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 22:01:51.709659 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.709637 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 22:01:51.757852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.757801 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:01:51.762610 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762567 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.762818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762640 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.762818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762746 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.762818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762785 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.762818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762809 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762830 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762856 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762880 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762900 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqwj5\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-kube-api-access-sqwj5\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762952 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-config-out\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.762992 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-config\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763031 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763054 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763081 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763105 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763132 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763172 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763196 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-web-config\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763244 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.763392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.763265 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.771132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771095 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 22:01:51.771379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771178 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 22:01:51.771379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771359 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 22:01:51.771496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771373 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 22:01:51.771496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771414 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 22:01:51.772064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771954 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 22:01:51.772064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771981 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hm895\"" Oct 30 22:01:51.772064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771993 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fq22m5ujhqugl\"" Oct 30 22:01:51.772064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.771982 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 22:01:51.772064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.772009 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 22:01:51.772064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.772030 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 22:01:51.864546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864506 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.864736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864564 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.864736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864601 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.864736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864632 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.864736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864655 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.864736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864685 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.864736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864715 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.864736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864737 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864757 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqwj5\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-kube-api-access-sqwj5\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864806 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-config-out\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864848 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-config\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864887 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864913 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864914 5226 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864934 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l6v6f" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864940 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864969 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.864999 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.865019 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.865039 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.865068 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-web-config\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.865104 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.865135 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.865749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.865731 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.866883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.866365 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.866883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.866560 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.867265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.867240 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.867992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.867973 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.868087 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.868070 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.868811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.868413 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.868811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.868453 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.868811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.868673 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.870163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.870115 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.870308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.870284 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-web-config\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.870893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.870863 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.871919 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.871896 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.872414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.872388 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-config-out\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.872946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.872911 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.873043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.872974 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.874416 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.874393 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-config\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.874509 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.874490 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.930875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.930846 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqwj5\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-kube-api-access-sqwj5\") pod \"prometheus-k8s-0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:51.967634 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.967599 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" event={"ID":"10f4cba6-57dc-4397-a602-21a3bad21b82","Type":"ContainerStarted","Data":"d6499bd2e0d381f6716fd16a1b60a7ec4eaf7bcb404a1b97f1eda64a8d855552"} Oct 30 22:01:51.968036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.967686 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" event={"ID":"10f4cba6-57dc-4397-a602-21a3bad21b82","Type":"ContainerStarted","Data":"56ac7b52375baf3ae25675da0ac78f3dd0bbe38b7ade4a997e1ce3307a1f7841"} Oct 30 22:01:51.968036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.967724 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" event={"ID":"10f4cba6-57dc-4397-a602-21a3bad21b82","Type":"ContainerStarted","Data":"1a994ee9f9911c7af8dc92a5aebbb17c66adf1cc3078c566703dae4f0260ff08"} Oct 30 22:01:51.975865 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:51.975820 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:52.117355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:52.117319 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6mdvj" Oct 30 22:01:52.420221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:52.419990 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:01:52.437690 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:01:52.430686 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440940dc_5688_4650_8edd_e3d9863e6aa0.slice/crio-27608c9322b5d7368567a547ddd6f27a00b643bd28090c020e4b895afb1f91f5 WatchSource:0}: Error finding container 27608c9322b5d7368567a547ddd6f27a00b643bd28090c020e4b895afb1f91f5: Status 404 returned error can't find the container with id 27608c9322b5d7368567a547ddd6f27a00b643bd28090c020e4b895afb1f91f5 Oct 30 22:01:52.975466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:52.974812 5226 generic.go:356] "Generic (PLEG): container finished" podID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" exitCode=0 Oct 30 22:01:52.975466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:52.975072 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerDied","Data":"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d"} Oct 30 22:01:52.975466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:52.975159 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerStarted","Data":"27608c9322b5d7368567a547ddd6f27a00b643bd28090c020e4b895afb1f91f5"} Oct 30 22:01:52.980199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:52.980168 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" event={"ID":"10f4cba6-57dc-4397-a602-21a3bad21b82","Type":"ContainerStarted","Data":"5ddf36b78783ff1f57cd4047308f72d257bcbeec9b3b32d63e61eed6ac713502"} Oct 30 22:01:52.980376 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:52.980259 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" event={"ID":"10f4cba6-57dc-4397-a602-21a3bad21b82","Type":"ContainerStarted","Data":"0aa0a6969a05a57e8ca430913d64838e2299e7107e8f758e9c2c46ccc876dd0b"} Oct 30 22:01:52.982897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:52.982869 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" event={"ID":"3058b5f8-ae0e-4193-b4dd-6d4abace3282","Type":"ContainerStarted","Data":"29feb845aa968bde5627b624319f6d6c9a2a88bc99bfd103253c90567462363f"} Oct 30 22:01:53.989704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.989672 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" event={"ID":"10f4cba6-57dc-4397-a602-21a3bad21b82","Type":"ContainerStarted","Data":"078a153592b85118f4d7440d984e11c6f2f9ae1799002b46722b1dfbbafddc4a"} Oct 30 22:01:53.990218 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.989816 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:01:53.993301 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.993265 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerStarted","Data":"fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42"} Oct 30 22:01:53.993484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.993350 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerStarted","Data":"29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100"} Oct 30 22:01:53.993484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.993390 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerStarted","Data":"70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9"} Oct 30 22:01:53.993484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.993418 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerStarted","Data":"795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab"} Oct 30 22:01:53.993484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.993452 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerStarted","Data":"ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e"} Oct 30 22:01:53.993665 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.993490 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerStarted","Data":"7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db"} Oct 30 22:01:53.995925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.995884 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" event={"ID":"9dce0456-dccb-4e0f-bdbd-978371813dc4","Type":"ContainerStarted","Data":"6baa67a4f66316fc0fe4f27087f7bdf6b51ab6dad7d9481268cb00d2bd8fa4f6"} Oct 30 22:01:53.996094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.995948 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" event={"ID":"9dce0456-dccb-4e0f-bdbd-978371813dc4","Type":"ContainerStarted","Data":"7ab918f6a64665eeaf97c233d0bccd31a222675d26f9e52adfe723d6224ca1b8"} Oct 30 22:01:53.996094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:53.995980 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc79ffd54-xnz4d" event={"ID":"9dce0456-dccb-4e0f-bdbd-978371813dc4","Type":"ContainerStarted","Data":"c5509d0ad07d95a7699d164607120132013ebcfabe2fb25b50f5bc934f7828ee"} Oct 30 22:01:54.419973 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:01:54.419881 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:01:54.527079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:54.527049 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4k4pr" Oct 30 22:01:57.010147 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:57.010108 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerStarted","Data":"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc"} Oct 30 22:01:57.010574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:57.010266 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerStarted","Data":"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d"} Oct 30 22:01:57.010574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:57.010310 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerStarted","Data":"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98"} Oct 30 22:01:57.010574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:57.010348 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerStarted","Data":"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d"} Oct 30 22:01:57.010574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:57.010383 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerStarted","Data":"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807"} Oct 30 22:01:58.017242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:58.017179 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerStarted","Data":"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41"} Oct 30 22:01:58.262837 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:58.262766 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.585219168 podStartE2EDuration="7.262750216s" totalImagesPullingTime="3.653897047s" totalInitContainerRuntime="23.634001ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:58.215897194 +0000 UTC m=+9.473238904" watchObservedRunningTime="2026-10-30 22:01:58.262750216 +0000 UTC m=+9.520091970" Oct 30 22:01:58.415917 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:58.415840 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-949b9f975-65pwb" Oct 30 22:01:58.793563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:58.793522 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:01:59.518185 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:59.518155 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:01:59.518604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:01:59.518433 5226 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 22:02:00.005688 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:00.005661 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-665bfc67b6-srdl5" Oct 30 22:02:03.289791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:03.289757 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlgss" Oct 30 22:02:06.809986 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:06.809946 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:02:08.291429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:08.291382 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:02:24.410670 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:02:24.410628 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:02:28.297317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:28.297282 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:02:28.301094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:28.301074 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-cbb75f5c9-k9gj6" Oct 30 22:02:46.107494 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:46.107448 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:02:46.126813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:46.126783 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:02:46.201361 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:46.201332 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:02:54.410717 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:02:54.410676 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:02:58.947656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:58.947616 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:02:58.948124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:58.947980 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy-metric" containerID="cri-o://29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100" gracePeriod=120 Oct 30 22:02:58.948124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:58.948018 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy-web" containerID="cri-o://795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab" gracePeriod=120 Oct 30 22:02:58.948124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:58.948047 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy" containerID="cri-o://70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9" gracePeriod=120 Oct 30 22:02:58.948124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:58.948071 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="prom-label-proxy" containerID="cri-o://fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42" gracePeriod=120 Oct 30 22:02:58.948124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:58.948020 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="config-reloader" containerID="cri-o://ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e" gracePeriod=120 Oct 30 22:02:58.948124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:58.947978 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="alertmanager" containerID="cri-o://7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db" gracePeriod=120 Oct 30 22:02:59.235589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235500 5226 generic.go:356] "Generic (PLEG): container finished" podID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerID="fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42" exitCode=0 Oct 30 22:02:59.235589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235527 5226 generic.go:356] "Generic (PLEG): container finished" podID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerID="29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100" exitCode=0 Oct 30 22:02:59.235589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235536 5226 generic.go:356] "Generic (PLEG): container finished" podID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerID="70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9" exitCode=0 Oct 30 22:02:59.235589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235542 5226 generic.go:356] "Generic (PLEG): container finished" podID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerID="ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e" exitCode=0 Oct 30 22:02:59.235589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235548 5226 generic.go:356] "Generic (PLEG): container finished" podID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerID="7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db" exitCode=0 Oct 30 22:02:59.235847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235584 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerDied","Data":"fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42"} Oct 30 22:02:59.235847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235652 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerDied","Data":"29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100"} Oct 30 22:02:59.235847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235682 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerDied","Data":"70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9"} Oct 30 22:02:59.235847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235705 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerDied","Data":"ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e"} Oct 30 22:02:59.235847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:02:59.235729 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerDied","Data":"7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db"} Oct 30 22:03:00.203423 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.203388 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.241855 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.241818 5226 generic.go:356] "Generic (PLEG): container finished" podID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerID="795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab" exitCode=0 Oct 30 22:03:00.242012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.241890 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerDied","Data":"795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab"} Oct 30 22:03:00.242012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.241914 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.242012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.241964 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffd5821b-7635-4bbf-9e07-86fa1579328d","Type":"ContainerDied","Data":"7552fcdfe513c4d4a5e6b2333faf9987c0c4359fa2ca9681c30eaa331d192049"} Oct 30 22:03:00.242012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.241999 5226 scope.go:122] "RemoveContainer" containerID="fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42" Oct 30 22:03:00.249638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.249619 5226 scope.go:122] "RemoveContainer" containerID="29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100" Oct 30 22:03:00.256854 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.256833 5226 scope.go:122] "RemoveContainer" containerID="70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9" Oct 30 22:03:00.263577 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.263555 5226 scope.go:122] "RemoveContainer" containerID="795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab" Oct 30 22:03:00.271280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.271255 5226 scope.go:122] "RemoveContainer" containerID="ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e" Oct 30 22:03:00.278696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.278674 5226 scope.go:122] "RemoveContainer" containerID="7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db" Oct 30 22:03:00.285731 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.285711 5226 scope.go:122] "RemoveContainer" containerID="8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0" Oct 30 22:03:00.292611 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.292591 5226 scope.go:122] "RemoveContainer" containerID="fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42" Oct 30 22:03:00.292915 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:00.292892 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42\": container with ID starting with fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42 not found: ID does not exist" containerID="fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42" Oct 30 22:03:00.293010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.292928 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42"} err="failed to get container status \"fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42\": rpc error: code = NotFound desc = could not find container \"fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42\": container with ID starting with fac57a74045ab97ff23e27932d6278452a69e56eb699134795ea6e2bd888dd42 not found: ID does not exist" Oct 30 22:03:00.293010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.292978 5226 scope.go:122] "RemoveContainer" containerID="29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100" Oct 30 22:03:00.293252 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:00.293233 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100\": container with ID starting with 29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100 not found: ID does not exist" containerID="29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100" Oct 30 22:03:00.293299 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.293261 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100"} err="failed to get container status \"29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100\": rpc error: code = NotFound desc = could not find container \"29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100\": container with ID starting with 29e381e32f98123a4c28850a62f4349d141c13ffd3e4f7e13ac015de021e0100 not found: ID does not exist" Oct 30 22:03:00.293299 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.293278 5226 scope.go:122] "RemoveContainer" containerID="70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9" Oct 30 22:03:00.293502 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:00.293484 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9\": container with ID starting with 70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9 not found: ID does not exist" containerID="70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9" Oct 30 22:03:00.293566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.293511 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9"} err="failed to get container status \"70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9\": rpc error: code = NotFound desc = could not find container \"70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9\": container with ID starting with 70359b0d0fdf6f61dbf5da46988742050ec0aca0b3c79035ce0843c720a968b9 not found: ID does not exist" Oct 30 22:03:00.293566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.293533 5226 scope.go:122] "RemoveContainer" containerID="795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab" Oct 30 22:03:00.293773 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:00.293756 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab\": container with ID starting with 795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab not found: ID does not exist" containerID="795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab" Oct 30 22:03:00.293813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.293779 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab"} err="failed to get container status \"795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab\": rpc error: code = NotFound desc = could not find container \"795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab\": container with ID starting with 795625844a003150c9023318073c4840e30336f5df5bdb7b8301f9da7eeab8ab not found: ID does not exist" Oct 30 22:03:00.293813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.293794 5226 scope.go:122] "RemoveContainer" containerID="ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e" Oct 30 22:03:00.294016 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:00.293999 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e\": container with ID starting with ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e not found: ID does not exist" containerID="ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e" Oct 30 22:03:00.294078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.294023 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e"} err="failed to get container status \"ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e\": rpc error: code = NotFound desc = could not find container \"ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e\": container with ID starting with ffdca48c607ccc526a5262274de2a79a9d5f46b9bf3bbdbccd351ac3cbfd419e not found: ID does not exist" Oct 30 22:03:00.294078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.294042 5226 scope.go:122] "RemoveContainer" containerID="7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db" Oct 30 22:03:00.294308 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:00.294292 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db\": container with ID starting with 7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db not found: ID does not exist" containerID="7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db" Oct 30 22:03:00.294350 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.294314 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db"} err="failed to get container status \"7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db\": rpc error: code = NotFound desc = could not find container \"7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db\": container with ID starting with 7dfe785fb772a6ea674aa2fd25f8203c0e662093756926b421394930501563db not found: ID does not exist" Oct 30 22:03:00.294350 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.294327 5226 scope.go:122] "RemoveContainer" containerID="8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0" Oct 30 22:03:00.294560 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:00.294543 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0\": container with ID starting with 8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0 not found: ID does not exist" containerID="8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0" Oct 30 22:03:00.294611 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.294569 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0"} err="failed to get container status \"8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0\": rpc error: code = NotFound desc = could not find container \"8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0\": container with ID starting with 8a38e493d12278a1dc98b9c133120182c465bd725a12d89caa794fcc5d360ec0 not found: ID does not exist" Oct 30 22:03:00.351808 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.351770 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.351808 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.351812 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.351847 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.351885 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.351925 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.351950 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352072 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352132 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352197 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352264 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352311 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352340 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:00.352525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352361 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352390 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls\") pod \"ffd5821b-7635-4bbf-9e07-86fa1579328d\" (UID: \"ffd5821b-7635-4bbf-9e07-86fa1579328d\") " Oct 30 22:03:00.352743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352680 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-metrics-client-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.352743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.352724 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:00.353409 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.353166 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:00.354983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.354960 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:00.356501 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.356472 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:00.356608 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.356538 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:00.356608 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.356560 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:00.356608 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.356594 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:00.356821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.356803 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:00.356991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.356970 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "kube-api-access-gprrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:00.357275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.357254 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:00.359823 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.359780 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:00.366780 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.366751 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config" pod "ffd5821b-7635-4bbf-9e07-86fa1579328d" (UID: "ffd5821b-7635-4bbf-9e07-86fa1579328d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:00.453468 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453430 5226 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453468 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453455 5226 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453468 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453466 5226 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453468 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453478 5226 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-out\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453488 5226 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffd5821b-7635-4bbf-9e07-86fa1579328d-alertmanager-main-db\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453497 5226 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-web-config\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453506 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gprrl\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-kube-api-access-gprrl\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453515 5226 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453524 5226 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-secret-alertmanager-main-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453535 5226 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-cluster-tls-config\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453543 5226 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffd5821b-7635-4bbf-9e07-86fa1579328d-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.453824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.453551 5226 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffd5821b-7635-4bbf-9e07-86fa1579328d-tls-assets\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:00.658525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.658489 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:03:00.663249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.663190 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:03:00.704492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704455 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704878 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="init-config-reloader" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704898 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="init-config-reloader" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704915 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="config-reloader" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704922 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="config-reloader" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704940 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="alertmanager" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704947 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="alertmanager" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704965 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy-web" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704972 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy-web" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704982 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704989 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.704999 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="prom-label-proxy" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705006 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="prom-label-proxy" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705018 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy-metric" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705025 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy-metric" Oct 30 22:03:00.705097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705096 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="config-reloader" Oct 30 22:03:00.705909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705108 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="alertmanager" Oct 30 22:03:00.705909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705120 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy-web" Oct 30 22:03:00.705909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705128 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy" Oct 30 22:03:00.705909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705136 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="prom-label-proxy" Oct 30 22:03:00.705909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.705145 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" containerName="kube-rbac-proxy-metric" Oct 30 22:03:00.709341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.709319 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.759056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759012 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 22:03:00.759299 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759279 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 22:03:00.759390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759303 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 22:03:00.759390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759356 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 22:03:00.759513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759431 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 22:03:00.759513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759437 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v5kds\"" Oct 30 22:03:00.759513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759455 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 22:03:00.759513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759481 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 22:03:00.759513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.759461 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 22:03:00.805671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.805590 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 22:03:00.806455 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.806434 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:03:00.857658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857622 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.857842 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857668 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.857842 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857725 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204af2f1-c066-4a89-9ddd-120ae692f6c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.857842 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857750 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/204af2f1-c066-4a89-9ddd-120ae692f6c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.857842 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857804 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/204af2f1-c066-4a89-9ddd-120ae692f6c2-config-out\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.857842 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857824 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.858002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857846 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.858002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857902 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.858002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.857935 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/204af2f1-c066-4a89-9ddd-120ae692f6c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.858093 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.858018 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/204af2f1-c066-4a89-9ddd-120ae692f6c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.858093 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.858048 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.858157 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.858107 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkhgn\" (UniqueName: \"kubernetes.io/projected/204af2f1-c066-4a89-9ddd-120ae692f6c2-kube-api-access-jkhgn\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.858157 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.858133 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-web-config\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.903153 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.903117 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd5821b-7635-4bbf-9e07-86fa1579328d" path="/var/lib/kubelet/pods/ffd5821b-7635-4bbf-9e07-86fa1579328d/volumes" Oct 30 22:03:00.958695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958662 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.958804 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958711 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.958804 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958743 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204af2f1-c066-4a89-9ddd-120ae692f6c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.958804 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958779 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/204af2f1-c066-4a89-9ddd-120ae692f6c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.958943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958810 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/204af2f1-c066-4a89-9ddd-120ae692f6c2-config-out\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.958943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958830 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.958943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958856 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.958943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958890 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.958943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958914 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/204af2f1-c066-4a89-9ddd-120ae692f6c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.959190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958948 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/204af2f1-c066-4a89-9ddd-120ae692f6c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.959190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.958979 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.959190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.959026 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkhgn\" (UniqueName: \"kubernetes.io/projected/204af2f1-c066-4a89-9ddd-120ae692f6c2-kube-api-access-jkhgn\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.959190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.959058 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-web-config\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.959944 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.959913 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/204af2f1-c066-4a89-9ddd-120ae692f6c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.960869 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.960594 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/204af2f1-c066-4a89-9ddd-120ae692f6c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.960869 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.960805 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/204af2f1-c066-4a89-9ddd-120ae692f6c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.961867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.961844 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/204af2f1-c066-4a89-9ddd-120ae692f6c2-config-out\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.962059 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.962033 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.962181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.962140 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.962277 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.962179 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/204af2f1-c066-4a89-9ddd-120ae692f6c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.962386 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.962368 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-web-config\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.962567 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.962546 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.962667 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.962650 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.963994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.963975 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:00.964076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:00.964036 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/204af2f1-c066-4a89-9ddd-120ae692f6c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:01.018840 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:01.018800 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkhgn\" (UniqueName: \"kubernetes.io/projected/204af2f1-c066-4a89-9ddd-120ae692f6c2-kube-api-access-jkhgn\") pod \"alertmanager-main-0\" (UID: \"204af2f1-c066-4a89-9ddd-120ae692f6c2\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:01.029732 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:01.029700 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 22:03:01.468308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:01.468275 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 22:03:01.474952 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:03:01.474921 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204af2f1_c066_4a89_9ddd_120ae692f6c2.slice/crio-4a7a3aa660da33ffe4d79423adbc5f5ce58a4fb65cd7107e44dda2de6b7bf673 WatchSource:0}: Error finding container 4a7a3aa660da33ffe4d79423adbc5f5ce58a4fb65cd7107e44dda2de6b7bf673: Status 404 returned error can't find the container with id 4a7a3aa660da33ffe4d79423adbc5f5ce58a4fb65cd7107e44dda2de6b7bf673 Oct 30 22:03:02.252402 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:02.252364 5226 generic.go:356] "Generic (PLEG): container finished" podID="204af2f1-c066-4a89-9ddd-120ae692f6c2" containerID="34094c9b484cd86447402eaa64be34f3cb84d98d98b8815e9f2881a8a9b36704" exitCode=0 Oct 30 22:03:02.252584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:02.252439 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"204af2f1-c066-4a89-9ddd-120ae692f6c2","Type":"ContainerDied","Data":"34094c9b484cd86447402eaa64be34f3cb84d98d98b8815e9f2881a8a9b36704"} Oct 30 22:03:02.252584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:02.252506 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"204af2f1-c066-4a89-9ddd-120ae692f6c2","Type":"ContainerStarted","Data":"4a7a3aa660da33ffe4d79423adbc5f5ce58a4fb65cd7107e44dda2de6b7bf673"} Oct 30 22:03:03.260275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:03.260236 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"204af2f1-c066-4a89-9ddd-120ae692f6c2","Type":"ContainerStarted","Data":"356cdc940c75c507832abda4262e158e6a26c3e776f0f6810779906f8cb95a23"} Oct 30 22:03:03.260747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:03.260307 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"204af2f1-c066-4a89-9ddd-120ae692f6c2","Type":"ContainerStarted","Data":"6f4fa8a7ee570746425e04c67352c96ab11dfd33bf433a0f7315078b062487d1"} Oct 30 22:03:03.260747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:03.260336 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"204af2f1-c066-4a89-9ddd-120ae692f6c2","Type":"ContainerStarted","Data":"1a2a95675fbbc6042c67ee68dba7c02a0c2aa0097a0b2abd2588ffd47083bdc5"} Oct 30 22:03:03.260747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:03.260359 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"204af2f1-c066-4a89-9ddd-120ae692f6c2","Type":"ContainerStarted","Data":"168f1e1ae19ed9a12fe1ed3932e7916d3bd549eac2a90244924d568ecf0d24c9"} Oct 30 22:03:03.260747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:03.260382 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"204af2f1-c066-4a89-9ddd-120ae692f6c2","Type":"ContainerStarted","Data":"a710e90acf72c2264354c87a147d9c8f9c72797a04fa6576264d6c194feb74c1"} Oct 30 22:03:03.260747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:03.260403 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"204af2f1-c066-4a89-9ddd-120ae692f6c2","Type":"ContainerStarted","Data":"908a78d2d36c0497f933c6ac820d4d12c55bc5384116b0017311fae54efd5023"} Oct 30 22:03:03.384725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:03.384660 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.373379577 podStartE2EDuration="3.384645783s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.266206ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:03.383699286 +0000 UTC m=+74.641041029" watchObservedRunningTime="2026-10-30 22:03:03.384645783 +0000 UTC m=+74.641987490" Oct 30 22:03:04.619379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.619341 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:03:04.619818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.619679 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="prometheus" containerID="cri-o://bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" gracePeriod=600 Oct 30 22:03:04.619818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.619723 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy-web" containerID="cri-o://4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" gracePeriod=600 Oct 30 22:03:04.619818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.619724 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" gracePeriod=600 Oct 30 22:03:04.619818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.619748 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy" containerID="cri-o://4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" gracePeriod=600 Oct 30 22:03:04.619818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.619767 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="thanos-sidecar" containerID="cri-o://2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" gracePeriod=600 Oct 30 22:03:04.620069 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.619857 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="config-reloader" containerID="cri-o://753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" gracePeriod=600 Oct 30 22:03:04.874351 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.874290 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:04.891215 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891177 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-grpc-tls\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891226 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-kube-rbac-proxy\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891252 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-config\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891271 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-tls-assets\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-tls-assets\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891299 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-metrics-client-certs\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891395 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-db\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891433 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891487 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-kube-api-access-sqwj5\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-kube-api-access-sqwj5\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891516 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891564 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-serving-certs-ca-bundle\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891597 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-kubelet-serving-ca-bundle\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891626 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-2\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891660 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-config-out\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891693 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-metrics-client-ca\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.891760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891736 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-thanos-prometheus-http-client-file\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.892050 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891781 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-tls\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.892050 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891812 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-1\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.892050 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891845 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-0\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.892050 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891884 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-trusted-ca-bundle\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.892050 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.891943 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-web-config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-web-config\") pod \"440940dc-5688-4650-8edd-e3d9863e6aa0\" (UID: \"440940dc-5688-4650-8edd-e3d9863e6aa0\") " Oct 30 22:03:04.893761 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.893729 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-metrics-client-ca" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:04.893889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.893730 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-grpc-tls" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.893971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.893906 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-1" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:04.893971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.893952 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-db" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:04.894935 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.894910 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-config" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.895023 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.894982 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-kube-rbac-proxy" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.895097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.895033 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-metrics-client-certs" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.895156 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.895091 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-tls-assets" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:04.895236 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.895172 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-2" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:04.895431 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.895403 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-trusted-ca-bundle" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:04.895588 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.895564 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-0" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:04.895675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.895571 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-serving-certs-ca-bundle" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:04.896017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.895993 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-kubelet-serving-ca-bundle" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:04.896361 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.896334 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-config-out" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:03:04.896445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.896426 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-tls" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.896500 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.896439 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-kube-rbac-proxy-web" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.896709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.896678 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-thanos-sidecar-tls" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.897561 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.897541 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-thanos-prometheus-http-client-file" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.897841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.897818 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-kube-api-access-sqwj5" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "kube-api-access-sqwj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:04.906681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.906658 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-web-config" pod "440940dc-5688-4650-8edd-e3d9863e6aa0" (UID: "440940dc-5688-4650-8edd-e3d9863e6aa0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:04.992870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992836 5226 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.992870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992868 5226 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.992870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992882 5226 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-web-config\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992895 5226 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-grpc-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992909 5226 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-kube-rbac-proxy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992922 5226 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-config\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992933 5226 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-tls-assets\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992945 5226 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-metrics-client-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992957 5226 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-db\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992970 5226 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.992987 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqwj5\" (UniqueName: \"kubernetes.io/projected/440940dc-5688-4650-8edd-e3d9863e6aa0-kube-api-access-sqwj5\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993000 5226 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993013 5226 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993026 5226 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993039 5226 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993054 5226 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/440940dc-5688-4650-8edd-e3d9863e6aa0-config-out\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993066 5226 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-configmap-metrics-client-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993078 5226 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993094 5226 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/440940dc-5688-4650-8edd-e3d9863e6aa0-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:04.993155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:04.993109 5226 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/440940dc-5688-4650-8edd-e3d9863e6aa0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:03:05.271384 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271347 5226 generic.go:356] "Generic (PLEG): container finished" podID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" exitCode=0 Oct 30 22:03:05.271384 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271377 5226 generic.go:356] "Generic (PLEG): container finished" podID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" exitCode=0 Oct 30 22:03:05.271384 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271385 5226 generic.go:356] "Generic (PLEG): container finished" podID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" exitCode=0 Oct 30 22:03:05.271384 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271391 5226 generic.go:356] "Generic (PLEG): container finished" podID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" exitCode=0 Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271396 5226 generic.go:356] "Generic (PLEG): container finished" podID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" exitCode=0 Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271400 5226 generic.go:356] "Generic (PLEG): container finished" podID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" exitCode=0 Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271420 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271420 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerDied","Data":"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41"} Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271549 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerDied","Data":"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc"} Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271577 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerDied","Data":"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d"} Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271603 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerDied","Data":"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98"} Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271629 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerDied","Data":"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d"} Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271635 5226 scope.go:122] "RemoveContainer" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271651 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerDied","Data":"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807"} Oct 30 22:03:05.271672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.271675 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"440940dc-5688-4650-8edd-e3d9863e6aa0","Type":"ContainerDied","Data":"27608c9322b5d7368567a547ddd6f27a00b643bd28090c020e4b895afb1f91f5"} Oct 30 22:03:05.282438 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.282422 5226 scope.go:122] "RemoveContainer" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" Oct 30 22:03:05.289216 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.289182 5226 scope.go:122] "RemoveContainer" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" Oct 30 22:03:05.295704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.295682 5226 scope.go:122] "RemoveContainer" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" Oct 30 22:03:05.302191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.302173 5226 scope.go:122] "RemoveContainer" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" Oct 30 22:03:05.308789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.308769 5226 scope.go:122] "RemoveContainer" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" Oct 30 22:03:05.315638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.315616 5226 scope.go:122] "RemoveContainer" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" Oct 30 22:03:05.322223 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.322186 5226 scope.go:122] "RemoveContainer" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" Oct 30 22:03:05.322514 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:05.322494 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": container with ID starting with c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41 not found: ID does not exist" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" Oct 30 22:03:05.322600 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.322529 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41"} err="failed to get container status \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": rpc error: code = NotFound desc = could not find container \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": container with ID starting with c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41 not found: ID does not exist" Oct 30 22:03:05.322600 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.322557 5226 scope.go:122] "RemoveContainer" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" Oct 30 22:03:05.322787 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:05.322770 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": container with ID starting with 4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc not found: ID does not exist" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" Oct 30 22:03:05.322856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.322795 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc"} err="failed to get container status \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": rpc error: code = NotFound desc = could not find container \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": container with ID starting with 4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc not found: ID does not exist" Oct 30 22:03:05.322856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.322815 5226 scope.go:122] "RemoveContainer" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" Oct 30 22:03:05.323080 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:05.323061 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": container with ID starting with 4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d not found: ID does not exist" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" Oct 30 22:03:05.323127 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323085 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d"} err="failed to get container status \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": rpc error: code = NotFound desc = could not find container \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": container with ID starting with 4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d not found: ID does not exist" Oct 30 22:03:05.323127 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323100 5226 scope.go:122] "RemoveContainer" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" Oct 30 22:03:05.323327 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:05.323310 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": container with ID starting with 2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98 not found: ID does not exist" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" Oct 30 22:03:05.323388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323336 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98"} err="failed to get container status \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": rpc error: code = NotFound desc = could not find container \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": container with ID starting with 2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98 not found: ID does not exist" Oct 30 22:03:05.323388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323357 5226 scope.go:122] "RemoveContainer" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" Oct 30 22:03:05.323550 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:05.323535 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": container with ID starting with 753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d not found: ID does not exist" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" Oct 30 22:03:05.323589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323553 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d"} err="failed to get container status \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": rpc error: code = NotFound desc = could not find container \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": container with ID starting with 753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d not found: ID does not exist" Oct 30 22:03:05.323589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323564 5226 scope.go:122] "RemoveContainer" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" Oct 30 22:03:05.323738 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:05.323724 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": container with ID starting with bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807 not found: ID does not exist" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" Oct 30 22:03:05.323775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323740 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807"} err="failed to get container status \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": rpc error: code = NotFound desc = could not find container \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": container with ID starting with bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807 not found: ID does not exist" Oct 30 22:03:05.323775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323761 5226 scope.go:122] "RemoveContainer" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" Oct 30 22:03:05.323966 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:05.323949 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": container with ID starting with 7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d not found: ID does not exist" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" Oct 30 22:03:05.324008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323969 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d"} err="failed to get container status \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": rpc error: code = NotFound desc = could not find container \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": container with ID starting with 7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d not found: ID does not exist" Oct 30 22:03:05.324008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.323982 5226 scope.go:122] "RemoveContainer" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" Oct 30 22:03:05.324167 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324152 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41"} err="failed to get container status \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": rpc error: code = NotFound desc = could not find container \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": container with ID starting with c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41 not found: ID does not exist" Oct 30 22:03:05.324492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324166 5226 scope.go:122] "RemoveContainer" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" Oct 30 22:03:05.324492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324393 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc"} err="failed to get container status \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": rpc error: code = NotFound desc = could not find container \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": container with ID starting with 4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc not found: ID does not exist" Oct 30 22:03:05.324492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324412 5226 scope.go:122] "RemoveContainer" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" Oct 30 22:03:05.324611 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324597 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d"} err="failed to get container status \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": rpc error: code = NotFound desc = could not find container \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": container with ID starting with 4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d not found: ID does not exist" Oct 30 22:03:05.324652 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324612 5226 scope.go:122] "RemoveContainer" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" Oct 30 22:03:05.324823 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324806 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98"} err="failed to get container status \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": rpc error: code = NotFound desc = could not find container \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": container with ID starting with 2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98 not found: ID does not exist" Oct 30 22:03:05.324869 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324823 5226 scope.go:122] "RemoveContainer" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" Oct 30 22:03:05.325007 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.324992 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d"} err="failed to get container status \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": rpc error: code = NotFound desc = could not find container \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": container with ID starting with 753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d not found: ID does not exist" Oct 30 22:03:05.325042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325007 5226 scope.go:122] "RemoveContainer" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" Oct 30 22:03:05.325190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325170 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807"} err="failed to get container status \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": rpc error: code = NotFound desc = could not find container \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": container with ID starting with bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807 not found: ID does not exist" Oct 30 22:03:05.325190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325186 5226 scope.go:122] "RemoveContainer" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" Oct 30 22:03:05.325420 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325405 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d"} err="failed to get container status \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": rpc error: code = NotFound desc = could not find container \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": container with ID starting with 7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d not found: ID does not exist" Oct 30 22:03:05.325420 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325419 5226 scope.go:122] "RemoveContainer" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" Oct 30 22:03:05.325614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325597 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41"} err="failed to get container status \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": rpc error: code = NotFound desc = could not find container \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": container with ID starting with c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41 not found: ID does not exist" Oct 30 22:03:05.325614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325612 5226 scope.go:122] "RemoveContainer" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" Oct 30 22:03:05.325808 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325791 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc"} err="failed to get container status \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": rpc error: code = NotFound desc = could not find container \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": container with ID starting with 4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc not found: ID does not exist" Oct 30 22:03:05.325808 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325806 5226 scope.go:122] "RemoveContainer" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" Oct 30 22:03:05.325997 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325982 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d"} err="failed to get container status \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": rpc error: code = NotFound desc = could not find container \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": container with ID starting with 4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d not found: ID does not exist" Oct 30 22:03:05.326048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.325997 5226 scope.go:122] "RemoveContainer" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" Oct 30 22:03:05.326190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326174 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98"} err="failed to get container status \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": rpc error: code = NotFound desc = could not find container \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": container with ID starting with 2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98 not found: ID does not exist" Oct 30 22:03:05.326190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326188 5226 scope.go:122] "RemoveContainer" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" Oct 30 22:03:05.326405 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326389 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d"} err="failed to get container status \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": rpc error: code = NotFound desc = could not find container \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": container with ID starting with 753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d not found: ID does not exist" Oct 30 22:03:05.326405 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326404 5226 scope.go:122] "RemoveContainer" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" Oct 30 22:03:05.326602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326586 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807"} err="failed to get container status \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": rpc error: code = NotFound desc = could not find container \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": container with ID starting with bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807 not found: ID does not exist" Oct 30 22:03:05.326602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326601 5226 scope.go:122] "RemoveContainer" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" Oct 30 22:03:05.326789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326773 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d"} err="failed to get container status \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": rpc error: code = NotFound desc = could not find container \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": container with ID starting with 7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d not found: ID does not exist" Oct 30 22:03:05.326789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326788 5226 scope.go:122] "RemoveContainer" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" Oct 30 22:03:05.326993 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326976 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41"} err="failed to get container status \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": rpc error: code = NotFound desc = could not find container \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": container with ID starting with c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41 not found: ID does not exist" Oct 30 22:03:05.326993 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.326992 5226 scope.go:122] "RemoveContainer" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" Oct 30 22:03:05.327191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.327176 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc"} err="failed to get container status \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": rpc error: code = NotFound desc = could not find container \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": container with ID starting with 4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc not found: ID does not exist" Oct 30 22:03:05.327248 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.327190 5226 scope.go:122] "RemoveContainer" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" Oct 30 22:03:05.327416 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.327398 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d"} err="failed to get container status \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": rpc error: code = NotFound desc = could not find container \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": container with ID starting with 4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d not found: ID does not exist" Oct 30 22:03:05.327466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.327416 5226 scope.go:122] "RemoveContainer" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" Oct 30 22:03:05.327636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.327617 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98"} err="failed to get container status \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": rpc error: code = NotFound desc = could not find container \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": container with ID starting with 2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98 not found: ID does not exist" Oct 30 22:03:05.327686 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.327637 5226 scope.go:122] "RemoveContainer" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" Oct 30 22:03:05.327837 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.327821 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d"} err="failed to get container status \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": rpc error: code = NotFound desc = could not find container \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": container with ID starting with 753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d not found: ID does not exist" Oct 30 22:03:05.327837 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.327836 5226 scope.go:122] "RemoveContainer" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" Oct 30 22:03:05.328044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328030 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807"} err="failed to get container status \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": rpc error: code = NotFound desc = could not find container \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": container with ID starting with bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807 not found: ID does not exist" Oct 30 22:03:05.328044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328043 5226 scope.go:122] "RemoveContainer" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" Oct 30 22:03:05.328283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328264 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d"} err="failed to get container status \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": rpc error: code = NotFound desc = could not find container \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": container with ID starting with 7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d not found: ID does not exist" Oct 30 22:03:05.328328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328284 5226 scope.go:122] "RemoveContainer" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" Oct 30 22:03:05.328493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328479 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41"} err="failed to get container status \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": rpc error: code = NotFound desc = could not find container \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": container with ID starting with c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41 not found: ID does not exist" Oct 30 22:03:05.328538 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328493 5226 scope.go:122] "RemoveContainer" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" Oct 30 22:03:05.328687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328667 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc"} err="failed to get container status \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": rpc error: code = NotFound desc = could not find container \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": container with ID starting with 4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc not found: ID does not exist" Oct 30 22:03:05.328742 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328688 5226 scope.go:122] "RemoveContainer" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" Oct 30 22:03:05.328898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328879 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d"} err="failed to get container status \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": rpc error: code = NotFound desc = could not find container \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": container with ID starting with 4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d not found: ID does not exist" Oct 30 22:03:05.328898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.328894 5226 scope.go:122] "RemoveContainer" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" Oct 30 22:03:05.329091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329071 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98"} err="failed to get container status \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": rpc error: code = NotFound desc = could not find container \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": container with ID starting with 2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98 not found: ID does not exist" Oct 30 22:03:05.329169 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329092 5226 scope.go:122] "RemoveContainer" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" Oct 30 22:03:05.329306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329282 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d"} err="failed to get container status \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": rpc error: code = NotFound desc = could not find container \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": container with ID starting with 753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d not found: ID does not exist" Oct 30 22:03:05.329306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329305 5226 scope.go:122] "RemoveContainer" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" Oct 30 22:03:05.329499 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329480 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807"} err="failed to get container status \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": rpc error: code = NotFound desc = could not find container \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": container with ID starting with bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807 not found: ID does not exist" Oct 30 22:03:05.329541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329499 5226 scope.go:122] "RemoveContainer" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" Oct 30 22:03:05.329686 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329669 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d"} err="failed to get container status \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": rpc error: code = NotFound desc = could not find container \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": container with ID starting with 7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d not found: ID does not exist" Oct 30 22:03:05.329730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329687 5226 scope.go:122] "RemoveContainer" containerID="c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41" Oct 30 22:03:05.329883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329867 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41"} err="failed to get container status \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": rpc error: code = NotFound desc = could not find container \"c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41\": container with ID starting with c453e56f8a8b6d6da339f10a2b5ae7dae55c4142b345bf994de9b7bb2f71cc41 not found: ID does not exist" Oct 30 22:03:05.329921 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.329882 5226 scope.go:122] "RemoveContainer" containerID="4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc" Oct 30 22:03:05.330077 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330062 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc"} err="failed to get container status \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": rpc error: code = NotFound desc = could not find container \"4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc\": container with ID starting with 4978877f48436357c698d2073c1eff698fdffc00fd56f5a50d411d12631924cc not found: ID does not exist" Oct 30 22:03:05.330119 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330077 5226 scope.go:122] "RemoveContainer" containerID="4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d" Oct 30 22:03:05.330300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330275 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d"} err="failed to get container status \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": rpc error: code = NotFound desc = could not find container \"4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d\": container with ID starting with 4ea74ee23b86cc40ad91e918932fad7beaed0052a53cf087446c7f74a119a56d not found: ID does not exist" Oct 30 22:03:05.330300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330294 5226 scope.go:122] "RemoveContainer" containerID="2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98" Oct 30 22:03:05.330513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330497 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98"} err="failed to get container status \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": rpc error: code = NotFound desc = could not find container \"2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98\": container with ID starting with 2ebf06b622b70bf99d02fed11a854815e65ce9252a2172b41656d83d62aa1d98 not found: ID does not exist" Oct 30 22:03:05.330568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330514 5226 scope.go:122] "RemoveContainer" containerID="753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d" Oct 30 22:03:05.330725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330708 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d"} err="failed to get container status \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": rpc error: code = NotFound desc = could not find container \"753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d\": container with ID starting with 753db56a9fc8d30de758e7859ce3c4b7b57d3c5757683a6453a4934c17d08f4d not found: ID does not exist" Oct 30 22:03:05.330761 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330727 5226 scope.go:122] "RemoveContainer" containerID="bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807" Oct 30 22:03:05.330925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330910 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807"} err="failed to get container status \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": rpc error: code = NotFound desc = could not find container \"bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807\": container with ID starting with bdb8b658af58f0c281fa1d4af40c70a73850d0e2e98c26afcdd2db1fe9e67807 not found: ID does not exist" Oct 30 22:03:05.330925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.330925 5226 scope.go:122] "RemoveContainer" containerID="7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d" Oct 30 22:03:05.331151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.331135 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d"} err="failed to get container status \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": rpc error: code = NotFound desc = could not find container \"7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d\": container with ID starting with 7bc2609521f426cc3ceacd28557f76a0a561d06feda6817affd22f712989c75d not found: ID does not exist" Oct 30 22:03:05.430009 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.429976 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:03:05.476742 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.476709 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:03:05.477191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477169 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:03:05.477488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477474 5226 cpu_manager.go:414] "containerMap: removing container" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy-web" Oct 30 22:03:05.477531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477490 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy-web" Oct 30 22:03:05.477531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477500 5226 cpu_manager.go:414] "containerMap: removing container" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy" Oct 30 22:03:05.477531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477505 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy" Oct 30 22:03:05.477531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477513 5226 cpu_manager.go:414] "containerMap: removing container" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="init-config-reloader" Oct 30 22:03:05.477531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477519 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="init-config-reloader" Oct 30 22:03:05.477531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477525 5226 cpu_manager.go:414] "containerMap: removing container" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="thanos-sidecar" Oct 30 22:03:05.477531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477530 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="thanos-sidecar" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477542 5226 cpu_manager.go:414] "containerMap: removing container" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="config-reloader" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477546 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="config-reloader" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477560 5226 cpu_manager.go:414] "containerMap: removing container" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="prometheus" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477565 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="prometheus" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477574 5226 cpu_manager.go:414] "containerMap: removing container" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy-thanos" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477579 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy-thanos" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477619 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="prometheus" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477627 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy-thanos" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477634 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy-web" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477638 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="kube-rbac-proxy" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477643 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="thanos-sidecar" Oct 30 22:03:05.477748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.477649 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" containerName="config-reloader" Oct 30 22:03:05.481800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.481779 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497427 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497401 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81841a12-765d-4fb9-b7e3-1c6414aaa29f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497435 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497459 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497518 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497547 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497565 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497619 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497669 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-config\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497743 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q688v\" (UniqueName: \"kubernetes.io/projected/81841a12-765d-4fb9-b7e3-1c6414aaa29f-kube-api-access-q688v\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497785 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497834 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497872 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497925 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-web-config\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.497942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497942 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.498096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.497964 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.498096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.498006 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.498096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.498039 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.498096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.498073 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81841a12-765d-4fb9-b7e3-1c6414aaa29f-config-out\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.498233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.498100 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.498233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.498118 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.533114 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.533023 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 22:03:05.580280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580245 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 22:03:05.580280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580249 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 22:03:05.580471 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580281 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 22:03:05.580724 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580707 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 22:03:05.580874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580857 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 22:03:05.580942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580915 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 22:03:05.580942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580927 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 22:03:05.581025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580936 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 22:03:05.581025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580936 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 22:03:05.581025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580983 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 22:03:05.581025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.580995 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 22:03:05.581182 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.581055 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fq22m5ujhqugl\"" Oct 30 22:03:05.581182 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.581169 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hm895\"" Oct 30 22:03:05.581327 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.581293 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 22:03:05.581382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.581367 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 22:03:05.581989 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.581972 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:03:05.598617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598593 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.598684 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598624 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-web-config\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.598684 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598643 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.598684 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598664 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.598816 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598795 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.598864 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598840 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.598916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598886 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81841a12-765d-4fb9-b7e3-1c6414aaa29f-config-out\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.598916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598908 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598929 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598962 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81841a12-765d-4fb9-b7e3-1c6414aaa29f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.598988 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599014 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599053 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599077 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599102 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599128 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599145 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599158 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-config\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599264 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q688v\" (UniqueName: \"kubernetes.io/projected/81841a12-765d-4fb9-b7e3-1c6414aaa29f-kube-api-access-q688v\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599304 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599313 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.599373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.599355 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.600800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.600774 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.601367 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.601339 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.601457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.601424 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-web-config\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.601540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.601518 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.601679 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.601656 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.601880 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.601858 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.602371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.602304 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.602371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.602308 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.602584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.602555 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.603092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.602988 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81841a12-765d-4fb9-b7e3-1c6414aaa29f-config-out\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.603092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.603046 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.603594 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.603548 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-config\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.603968 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.603943 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.604400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.604373 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.604541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.604523 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81841a12-765d-4fb9-b7e3-1c6414aaa29f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.604925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.604907 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81841a12-765d-4fb9-b7e3-1c6414aaa29f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.605620 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.605599 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81841a12-765d-4fb9-b7e3-1c6414aaa29f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.658346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.658316 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q688v\" (UniqueName: \"kubernetes.io/projected/81841a12-765d-4fb9-b7e3-1c6414aaa29f-kube-api-access-q688v\") pod \"prometheus-k8s-0\" (UID: \"81841a12-765d-4fb9-b7e3-1c6414aaa29f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:05.801632 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:05.801541 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:06.230774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:06.230741 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 22:03:06.238487 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:03:06.238456 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81841a12_765d_4fb9_b7e3_1c6414aaa29f.slice/crio-52f815e14c02541f726008717a7abc21f9f595e0ae4e0a65e7dcd313fcab5006 WatchSource:0}: Error finding container 52f815e14c02541f726008717a7abc21f9f595e0ae4e0a65e7dcd313fcab5006: Status 404 returned error can't find the container with id 52f815e14c02541f726008717a7abc21f9f595e0ae4e0a65e7dcd313fcab5006 Oct 30 22:03:06.277516 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:06.277484 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81841a12-765d-4fb9-b7e3-1c6414aaa29f","Type":"ContainerStarted","Data":"52f815e14c02541f726008717a7abc21f9f595e0ae4e0a65e7dcd313fcab5006"} Oct 30 22:03:06.902669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:06.902637 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440940dc-5688-4650-8edd-e3d9863e6aa0" path="/var/lib/kubelet/pods/440940dc-5688-4650-8edd-e3d9863e6aa0/volumes" Oct 30 22:03:07.282135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:07.282096 5226 generic.go:356] "Generic (PLEG): container finished" podID="81841a12-765d-4fb9-b7e3-1c6414aaa29f" containerID="502491207f17c9c073ef4d1ed5f2ce96ee599290edb4fc6cc516a23e5a6c8f1b" exitCode=0 Oct 30 22:03:07.282284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:07.282139 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81841a12-765d-4fb9-b7e3-1c6414aaa29f","Type":"ContainerDied","Data":"502491207f17c9c073ef4d1ed5f2ce96ee599290edb4fc6cc516a23e5a6c8f1b"} Oct 30 22:03:08.289073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:08.289039 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81841a12-765d-4fb9-b7e3-1c6414aaa29f","Type":"ContainerStarted","Data":"e316b9011b367426465bed496fbf833c1aa13999294c7588dc3a11566f4cab17"} Oct 30 22:03:08.289574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:08.289133 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81841a12-765d-4fb9-b7e3-1c6414aaa29f","Type":"ContainerStarted","Data":"9e1069de48bf422c82c90e39e80fa68475e7f7c89bd1f0dfcc42f127bf4cbdfd"} Oct 30 22:03:08.289574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:08.289162 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81841a12-765d-4fb9-b7e3-1c6414aaa29f","Type":"ContainerStarted","Data":"53e840b69e2df077a6c8f67590e4a2058583891f5781ee3957dd8e775295097d"} Oct 30 22:03:08.289574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:08.289186 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81841a12-765d-4fb9-b7e3-1c6414aaa29f","Type":"ContainerStarted","Data":"8b179243dafdc05a5d7e78d3f215a4dc4404f8bac880f758a8eb1669e0af10d3"} Oct 30 22:03:08.289574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:08.289237 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81841a12-765d-4fb9-b7e3-1c6414aaa29f","Type":"ContainerStarted","Data":"5bd8ba290b56ea389a8701a094df97eddbc3018b18d355dfbee61e002a7dc8be"} Oct 30 22:03:08.289574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:08.289270 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81841a12-765d-4fb9-b7e3-1c6414aaa29f","Type":"ContainerStarted","Data":"290a06c8df4133c3530596101ee87309be3b82740896e7ee52cc44f64b1021af"} Oct 30 22:03:08.416648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:08.416595 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.399619785 podStartE2EDuration="3.416580957s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.961172ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:08.41470148 +0000 UTC m=+79.672043191" watchObservedRunningTime="2026-10-30 22:03:08.416580957 +0000 UTC m=+79.673922667" Oct 30 22:03:10.802566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:03:10.802526 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:03:24.409746 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:24.409709 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:03:54.409695 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:03:54.409650 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:04:05.802516 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:04:05.802468 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:04:05.817797 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:04:05.817772 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:04:06.510936 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:04:06.510910 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 22:04:24.409428 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:04:24.409391 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:04:54.408895 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:04:54.408856 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:05:24.409315 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:05:24.409235 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:05:36.220172 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.220137 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz"] Oct 30 22:05:36.222339 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.222323 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.272719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.272690 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 22:05:36.272897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.272724 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 22:05:36.290557 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.290535 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 22:05:36.319164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.319121 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8ctw4\"" Oct 30 22:05:36.320040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.320019 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz"] Oct 30 22:05:36.353090 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353059 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztjt\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-kube-api-access-4ztjt\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.353303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353104 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.353303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353128 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.353303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353197 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a989905-c9be-4423-8d8c-83400f966607-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.353303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353281 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.353529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353380 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.353529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353414 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.353529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353431 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a989905-c9be-4423-8d8c-83400f966607-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.353529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.353493 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.454937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.454905 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.454937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.454936 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a989905-c9be-4423-8d8c-83400f966607-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.454967 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.454992 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455013 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455033 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a989905-c9be-4423-8d8c-83400f966607-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455065 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455120 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztjt\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-kube-api-access-4ztjt\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455156 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455440 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455486 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455619 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455623 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.455994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.455800 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a989905-c9be-4423-8d8c-83400f966607-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.457420 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.457396 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.457722 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.457706 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a989905-c9be-4423-8d8c-83400f966607-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.512749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.512676 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.512889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.512863 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztjt\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-kube-api-access-4ztjt\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nmwcz\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.533662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.533637 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:36.956671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:36.956528 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz"] Oct 30 22:05:36.967285 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:05:36.964151 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a989905_c9be_4423_8d8c_83400f966607.slice/crio-da46e8248b02f9bc9f9a2599b263208313cdab48f2faec7f2211b44a46822930 WatchSource:0}: Error finding container da46e8248b02f9bc9f9a2599b263208313cdab48f2faec7f2211b44a46822930: Status 404 returned error can't find the container with id da46e8248b02f9bc9f9a2599b263208313cdab48f2faec7f2211b44a46822930 Oct 30 22:05:37.808877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:37.808840 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" event={"ID":"3a989905-c9be-4423-8d8c-83400f966607","Type":"ContainerStarted","Data":"da46e8248b02f9bc9f9a2599b263208313cdab48f2faec7f2211b44a46822930"} Oct 30 22:05:38.039075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.039034 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44"] Oct 30 22:05:38.041906 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.041883 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.071088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.071014 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv"] Oct 30 22:05:38.073355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.073334 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.116767 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.116737 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 22:05:38.116926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.116785 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-7wgp9\"" Oct 30 22:05:38.116926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.116863 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 22:05:38.138030 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.138001 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn"] Oct 30 22:05:38.140690 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.140673 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44"] Oct 30 22:05:38.140778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.140766 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.164710 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.164680 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv"] Oct 30 22:05:38.166628 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.166604 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn"] Oct 30 22:05:38.169718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.169687 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwnb\" (UniqueName: \"kubernetes.io/projected/cacda298-e36a-4f01-81f7-9e3fa68a66a1-kube-api-access-kdwnb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.169838 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.169741 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.169905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.169861 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.169905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.169894 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.170019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.169978 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.170019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.170005 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmkt\" (UniqueName: \"kubernetes.io/projected/92af0346-7bab-4c9f-8a62-a5146ff1fa35-kube-api-access-lwmkt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.194906 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.194880 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-fsj7d\"" Oct 30 22:05:38.195048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.194920 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xbdd4\"" Oct 30 22:05:38.270779 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.270725 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwnb\" (UniqueName: \"kubernetes.io/projected/cacda298-e36a-4f01-81f7-9e3fa68a66a1-kube-api-access-kdwnb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.270779 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.270779 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.270995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.270813 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.270995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.270831 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.270995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.270859 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.270995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.270933 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmkt\" (UniqueName: \"kubernetes.io/projected/92af0346-7bab-4c9f-8a62-a5146ff1fa35-kube-api-access-lwmkt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.270995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.270977 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.271247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.271022 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.271247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.271071 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2k4\" (UniqueName: \"kubernetes.io/projected/21053dee-5d7f-490c-9789-abf59b979e71-kube-api-access-gg2k4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.271396 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.271302 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.271396 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.271309 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.271396 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.271309 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.271572 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.271445 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.329033 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.328959 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmkt\" (UniqueName: \"kubernetes.io/projected/92af0346-7bab-4c9f-8a62-a5146ff1fa35-kube-api-access-lwmkt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.329033 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.328960 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwnb\" (UniqueName: \"kubernetes.io/projected/cacda298-e36a-4f01-81f7-9e3fa68a66a1-kube-api-access-kdwnb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.355952 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.355926 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:05:38.371759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.371726 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.371865 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.371777 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2k4\" (UniqueName: \"kubernetes.io/projected/21053dee-5d7f-490c-9789-abf59b979e71-kube-api-access-gg2k4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.371865 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.371857 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.372164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.372142 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.372164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.372159 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.386891 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.386864 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:05:38.432563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.432532 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2k4\" (UniqueName: \"kubernetes.io/projected/21053dee-5d7f-490c-9789-abf59b979e71-kube-api-access-gg2k4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.454549 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.454526 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:05:38.758271 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.758235 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44"] Oct 30 22:05:38.765158 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:05:38.765122 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92af0346_7bab_4c9f_8a62_a5146ff1fa35.slice/crio-eb0bc252221e7e03f025b3234dedec7c0fe7f515e73e4285d816dbcf88559e14 WatchSource:0}: Error finding container eb0bc252221e7e03f025b3234dedec7c0fe7f515e73e4285d816dbcf88559e14: Status 404 returned error can't find the container with id eb0bc252221e7e03f025b3234dedec7c0fe7f515e73e4285d816dbcf88559e14 Oct 30 22:05:38.772759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.772734 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv"] Oct 30 22:05:38.779114 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:05:38.779088 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacda298_e36a_4f01_81f7_9e3fa68a66a1.slice/crio-66929101d5e305f2d2f15eb4619926aa005b6c9ca09b58c99cce5855d99728d9 WatchSource:0}: Error finding container 66929101d5e305f2d2f15eb4619926aa005b6c9ca09b58c99cce5855d99728d9: Status 404 returned error can't find the container with id 66929101d5e305f2d2f15eb4619926aa005b6c9ca09b58c99cce5855d99728d9 Oct 30 22:05:38.813689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.813661 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" event={"ID":"cacda298-e36a-4f01-81f7-9e3fa68a66a1","Type":"ContainerStarted","Data":"66929101d5e305f2d2f15eb4619926aa005b6c9ca09b58c99cce5855d99728d9"} Oct 30 22:05:38.814730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.814669 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" event={"ID":"92af0346-7bab-4c9f-8a62-a5146ff1fa35","Type":"ContainerStarted","Data":"eb0bc252221e7e03f025b3234dedec7c0fe7f515e73e4285d816dbcf88559e14"} Oct 30 22:05:38.825526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:38.825501 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn"] Oct 30 22:05:38.832768 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:05:38.832748 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21053dee_5d7f_490c_9789_abf59b979e71.slice/crio-7a908b21e506081f2b2b01adf4743d9089ddff3b8316c1040a2de60fcd3f25c3 WatchSource:0}: Error finding container 7a908b21e506081f2b2b01adf4743d9089ddff3b8316c1040a2de60fcd3f25c3: Status 404 returned error can't find the container with id 7a908b21e506081f2b2b01adf4743d9089ddff3b8316c1040a2de60fcd3f25c3 Oct 30 22:05:39.820745 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:39.820706 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" event={"ID":"21053dee-5d7f-490c-9789-abf59b979e71","Type":"ContainerStarted","Data":"7a908b21e506081f2b2b01adf4743d9089ddff3b8316c1040a2de60fcd3f25c3"} Oct 30 22:05:41.517476 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:41.517440 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:05:41.517962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:41.517533 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:05:41.517962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:41.517580 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:05:41.834696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:41.834603 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" event={"ID":"3a989905-c9be-4423-8d8c-83400f966607","Type":"ContainerStarted","Data":"55558ea82e0caff68e3ca06b526f67413f279e8fd002004e31d77e2ec2c62aff"} Oct 30 22:05:41.996631 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:41.996563 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" podStartSLOduration=1.446176513 podStartE2EDuration="5.996545246s" totalImagesPullingTime="4.550368733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:41.995427713 +0000 UTC m=+233.252769467" watchObservedRunningTime="2026-10-30 22:05:41.996545246 +0000 UTC m=+233.253886959" Oct 30 22:05:42.534180 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:42.534144 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:42.539484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:42.539459 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:42.841234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:42.841138 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:42.842306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:42.842278 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:05:47.866164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:47.866123 5226 generic.go:356] "Generic (PLEG): container finished" podID="21053dee-5d7f-490c-9789-abf59b979e71" containerID="9f5a13a24c943bbdb3bebd030fc6d41e26551b0de0d46434651fd518f40ab009" exitCode=0 Oct 30 22:05:47.866671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:47.866224 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" event={"ID":"21053dee-5d7f-490c-9789-abf59b979e71","Type":"ContainerDied","Data":"9f5a13a24c943bbdb3bebd030fc6d41e26551b0de0d46434651fd518f40ab009"} Oct 30 22:05:48.871117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:48.871080 5226 generic.go:356] "Generic (PLEG): container finished" podID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerID="f19adbe97b17c65b1646d15a99ffdffd56bffc87f787376c91ea0cc0d777f9a5" exitCode=0 Oct 30 22:05:48.871603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:48.871156 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" event={"ID":"cacda298-e36a-4f01-81f7-9e3fa68a66a1","Type":"ContainerDied","Data":"f19adbe97b17c65b1646d15a99ffdffd56bffc87f787376c91ea0cc0d777f9a5"} Oct 30 22:05:48.872740 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:48.872715 5226 generic.go:356] "Generic (PLEG): container finished" podID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerID="98eb7913451fe1183686ef14f0c53a419eb6c28bfd9ba81db4cbff3ce6d6eea3" exitCode=0 Oct 30 22:05:48.872860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:48.872802 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" event={"ID":"92af0346-7bab-4c9f-8a62-a5146ff1fa35","Type":"ContainerDied","Data":"98eb7913451fe1183686ef14f0c53a419eb6c28bfd9ba81db4cbff3ce6d6eea3"} Oct 30 22:05:49.877885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:49.877794 5226 generic.go:356] "Generic (PLEG): container finished" podID="21053dee-5d7f-490c-9789-abf59b979e71" containerID="9d12e3524f3302528d0918bfcbabe9ce413062b0035de0858b3b2178051baddf" exitCode=0 Oct 30 22:05:49.878288 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:49.877877 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" event={"ID":"21053dee-5d7f-490c-9789-abf59b979e71","Type":"ContainerDied","Data":"9d12e3524f3302528d0918bfcbabe9ce413062b0035de0858b3b2178051baddf"} Oct 30 22:05:50.883710 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:50.883677 5226 generic.go:356] "Generic (PLEG): container finished" podID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerID="0b9c5347f21d21b24b376ee55f0043bcb5b4e501f87bfaabc60bbe888042b3dd" exitCode=0 Oct 30 22:05:50.884137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:50.883764 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" event={"ID":"cacda298-e36a-4f01-81f7-9e3fa68a66a1","Type":"ContainerDied","Data":"0b9c5347f21d21b24b376ee55f0043bcb5b4e501f87bfaabc60bbe888042b3dd"} Oct 30 22:05:50.885786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:50.885765 5226 generic.go:356] "Generic (PLEG): container finished" podID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerID="b63d54bad1a5bd55f0fb6607f74715f13ecfd61aa0a9b1a46228f37e296c026b" exitCode=0 Oct 30 22:05:50.885845 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:50.885819 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" event={"ID":"92af0346-7bab-4c9f-8a62-a5146ff1fa35","Type":"ContainerDied","Data":"b63d54bad1a5bd55f0fb6607f74715f13ecfd61aa0a9b1a46228f37e296c026b"} Oct 30 22:05:54.422363 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:05:54.422321 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:05:58.928533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:58.928492 5226 generic.go:356] "Generic (PLEG): container finished" podID="21053dee-5d7f-490c-9789-abf59b979e71" containerID="e315f4358c287983e20325d70a502d511de852a2dfb3fd40e1b6069caa779ccc" exitCode=0 Oct 30 22:05:58.929045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:05:58.928571 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" event={"ID":"21053dee-5d7f-490c-9789-abf59b979e71","Type":"ContainerDied","Data":"e315f4358c287983e20325d70a502d511de852a2dfb3fd40e1b6069caa779ccc"} Oct 30 22:06:00.076254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.076230 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:06:00.178111 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.178081 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-util\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-util\") pod \"21053dee-5d7f-490c-9789-abf59b979e71\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " Oct 30 22:06:00.178306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.178199 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21053dee-5d7f-490c-9789-abf59b979e71-kube-api-access-gg2k4\" (UniqueName: \"kubernetes.io/projected/21053dee-5d7f-490c-9789-abf59b979e71-kube-api-access-gg2k4\") pod \"21053dee-5d7f-490c-9789-abf59b979e71\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " Oct 30 22:06:00.178306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.178277 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-bundle\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-bundle\") pod \"21053dee-5d7f-490c-9789-abf59b979e71\" (UID: \"21053dee-5d7f-490c-9789-abf59b979e71\") " Oct 30 22:06:00.178735 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.178713 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-bundle" pod "21053dee-5d7f-490c-9789-abf59b979e71" (UID: "21053dee-5d7f-490c-9789-abf59b979e71"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:00.180546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.180517 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21053dee-5d7f-490c-9789-abf59b979e71-kube-api-access-gg2k4" pod "21053dee-5d7f-490c-9789-abf59b979e71" (UID: "21053dee-5d7f-490c-9789-abf59b979e71"). InnerVolumeSpecName "kube-api-access-gg2k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:00.183028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.182976 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-util" pod "21053dee-5d7f-490c-9789-abf59b979e71" (UID: "21053dee-5d7f-490c-9789-abf59b979e71"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:00.279635 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.279597 5226 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-util\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:00.279635 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.279637 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg2k4\" (UniqueName: \"kubernetes.io/projected/21053dee-5d7f-490c-9789-abf59b979e71-kube-api-access-gg2k4\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:00.279841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.279653 5226 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21053dee-5d7f-490c-9789-abf59b979e71-bundle\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:00.938355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.938314 5226 generic.go:356] "Generic (PLEG): container finished" podID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerID="d33f1eab6ed356c3080598a22b8aa015f8570d3e28bac10b762b6f785dc21afc" exitCode=0 Oct 30 22:06:00.938498 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.938384 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" event={"ID":"cacda298-e36a-4f01-81f7-9e3fa68a66a1","Type":"ContainerDied","Data":"d33f1eab6ed356c3080598a22b8aa015f8570d3e28bac10b762b6f785dc21afc"} Oct 30 22:06:00.940487 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.940458 5226 generic.go:356] "Generic (PLEG): container finished" podID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerID="b183e2e91eaeae1cd02abdc004601834b6a4495494cd60641cd5de1e1605627f" exitCode=0 Oct 30 22:06:00.940597 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.940540 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" event={"ID":"92af0346-7bab-4c9f-8a62-a5146ff1fa35","Type":"ContainerDied","Data":"b183e2e91eaeae1cd02abdc004601834b6a4495494cd60641cd5de1e1605627f"} Oct 30 22:06:00.942187 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.942167 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" event={"ID":"21053dee-5d7f-490c-9789-abf59b979e71","Type":"ContainerDied","Data":"7a908b21e506081f2b2b01adf4743d9089ddff3b8316c1040a2de60fcd3f25c3"} Oct 30 22:06:00.942272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.942199 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgvzcn" Oct 30 22:06:00.942272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:00.942219 5226 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a908b21e506081f2b2b01adf4743d9089ddff3b8316c1040a2de60fcd3f25c3" Oct 30 22:06:02.091430 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.091405 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:06:02.095595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.095573 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:06:02.196586 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.196544 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-util\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-util\") pod \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " Oct 30 22:06:02.196586 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.196592 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-bundle\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-bundle\") pod \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " Oct 30 22:06:02.196830 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.196639 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92af0346-7bab-4c9f-8a62-a5146ff1fa35-kube-api-access-lwmkt\" (UniqueName: \"kubernetes.io/projected/92af0346-7bab-4c9f-8a62-a5146ff1fa35-kube-api-access-lwmkt\") pod \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\" (UID: \"92af0346-7bab-4c9f-8a62-a5146ff1fa35\") " Oct 30 22:06:02.196830 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.196674 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-bundle\") pod \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " Oct 30 22:06:02.196830 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.196704 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-util\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-util\") pod \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " Oct 30 22:06:02.196830 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.196738 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacda298-e36a-4f01-81f7-9e3fa68a66a1-kube-api-access-kdwnb\" (UniqueName: \"kubernetes.io/projected/cacda298-e36a-4f01-81f7-9e3fa68a66a1-kube-api-access-kdwnb\") pod \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\" (UID: \"cacda298-e36a-4f01-81f7-9e3fa68a66a1\") " Oct 30 22:06:02.197196 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.197165 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-bundle" pod "92af0346-7bab-4c9f-8a62-a5146ff1fa35" (UID: "92af0346-7bab-4c9f-8a62-a5146ff1fa35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.197436 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.197411 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-bundle" pod "cacda298-e36a-4f01-81f7-9e3fa68a66a1" (UID: "cacda298-e36a-4f01-81f7-9e3fa68a66a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.198747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.198720 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92af0346-7bab-4c9f-8a62-a5146ff1fa35-kube-api-access-lwmkt" pod "92af0346-7bab-4c9f-8a62-a5146ff1fa35" (UID: "92af0346-7bab-4c9f-8a62-a5146ff1fa35"). InnerVolumeSpecName "kube-api-access-lwmkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:02.198834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.198815 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacda298-e36a-4f01-81f7-9e3fa68a66a1-kube-api-access-kdwnb" pod "cacda298-e36a-4f01-81f7-9e3fa68a66a1" (UID: "cacda298-e36a-4f01-81f7-9e3fa68a66a1"). InnerVolumeSpecName "kube-api-access-kdwnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:02.201355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.201331 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-util" pod "cacda298-e36a-4f01-81f7-9e3fa68a66a1" (UID: "cacda298-e36a-4f01-81f7-9e3fa68a66a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.204642 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.204613 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-util" pod "92af0346-7bab-4c9f-8a62-a5146ff1fa35" (UID: "92af0346-7bab-4c9f-8a62-a5146ff1fa35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.297955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.297929 5226 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-util\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.297955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.297950 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwnb\" (UniqueName: \"kubernetes.io/projected/cacda298-e36a-4f01-81f7-9e3fa68a66a1-kube-api-access-kdwnb\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.297955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.297960 5226 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-util\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.298139 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.297972 5226 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92af0346-7bab-4c9f-8a62-a5146ff1fa35-bundle\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.298139 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.297981 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwmkt\" (UniqueName: \"kubernetes.io/projected/92af0346-7bab-4c9f-8a62-a5146ff1fa35-kube-api-access-lwmkt\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.298139 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.297993 5226 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cacda298-e36a-4f01-81f7-9e3fa68a66a1-bundle\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.952151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.952121 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" Oct 30 22:06:02.952326 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.952148 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7l7n44" event={"ID":"92af0346-7bab-4c9f-8a62-a5146ff1fa35","Type":"ContainerDied","Data":"eb0bc252221e7e03f025b3234dedec7c0fe7f515e73e4285d816dbcf88559e14"} Oct 30 22:06:02.952326 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.952223 5226 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb0bc252221e7e03f025b3234dedec7c0fe7f515e73e4285d816dbcf88559e14" Oct 30 22:06:02.954164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.954134 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" event={"ID":"cacda298-e36a-4f01-81f7-9e3fa68a66a1","Type":"ContainerDied","Data":"66929101d5e305f2d2f15eb4619926aa005b6c9ca09b58c99cce5855d99728d9"} Oct 30 22:06:02.954308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.954180 5226 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66929101d5e305f2d2f15eb4619926aa005b6c9ca09b58c99cce5855d99728d9" Oct 30 22:06:02.954308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:02.954269 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4rpgv" Oct 30 22:06:13.620043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620008 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s"] Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620373 5226 cpu_manager.go:414] "containerMap: removing container" podUID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerName="util" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620387 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerName="util" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620404 5226 cpu_manager.go:414] "containerMap: removing container" podUID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerName="pull" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620409 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerName="pull" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620420 5226 cpu_manager.go:414] "containerMap: removing container" podUID="21053dee-5d7f-490c-9789-abf59b979e71" containerName="util" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620425 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21053dee-5d7f-490c-9789-abf59b979e71" containerName="util" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620431 5226 cpu_manager.go:414] "containerMap: removing container" podUID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerName="util" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620437 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerName="util" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620442 5226 cpu_manager.go:414] "containerMap: removing container" podUID="21053dee-5d7f-490c-9789-abf59b979e71" containerName="extract" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620447 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21053dee-5d7f-490c-9789-abf59b979e71" containerName="extract" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620453 5226 cpu_manager.go:414] "containerMap: removing container" podUID="21053dee-5d7f-490c-9789-abf59b979e71" containerName="pull" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620465 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21053dee-5d7f-490c-9789-abf59b979e71" containerName="pull" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620482 5226 cpu_manager.go:414] "containerMap: removing container" podUID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerName="extract" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620487 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerName="extract" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620503 5226 cpu_manager.go:414] "containerMap: removing container" podUID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerName="extract" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620508 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerName="extract" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620515 5226 cpu_manager.go:414] "containerMap: removing container" podUID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerName="pull" Oct 30 22:06:13.620543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620519 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerName="pull" Oct 30 22:06:13.621060 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620571 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="21053dee-5d7f-490c-9789-abf59b979e71" containerName="extract" Oct 30 22:06:13.621060 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620580 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacda298-e36a-4f01-81f7-9e3fa68a66a1" containerName="extract" Oct 30 22:06:13.621060 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.620589 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="92af0346-7bab-4c9f-8a62-a5146ff1fa35" containerName="extract" Oct 30 22:06:13.623527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.623512 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.692648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.692615 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a384e39c-657d-4e8c-8c85-d41b60332868-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.692648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.692650 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.692878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.692768 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.692878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.692847 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.692975 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.692876 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppdx\" (UniqueName: \"kubernetes.io/projected/a384e39c-657d-4e8c-8c85-d41b60332868-kube-api-access-2ppdx\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.692975 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.692909 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a384e39c-657d-4e8c-8c85-d41b60332868-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.719147 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.719110 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 22:06:13.719334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.719168 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 22:06:13.719334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.719197 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 22:06:13.719334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.719229 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 22:06:13.719334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.719244 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8hr44\"" Oct 30 22:06:13.719334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.719279 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 22:06:13.793872 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.793827 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.793872 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.793874 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.794159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.793900 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppdx\" (UniqueName: \"kubernetes.io/projected/a384e39c-657d-4e8c-8c85-d41b60332868-kube-api-access-2ppdx\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.794159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.793940 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a384e39c-657d-4e8c-8c85-d41b60332868-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.794159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.793971 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a384e39c-657d-4e8c-8c85-d41b60332868-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.794159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.793998 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.794988 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.794412 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a384e39c-657d-4e8c-8c85-d41b60332868-manifests\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.796837 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.796812 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a384e39c-657d-4e8c-8c85-d41b60332868-tmp\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.796945 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.796859 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-metrics-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.797055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.797036 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-webhook-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.797128 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.797108 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a384e39c-657d-4e8c-8c85-d41b60332868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.852762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.852728 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppdx\" (UniqueName: \"kubernetes.io/projected/a384e39c-657d-4e8c-8c85-d41b60332868-kube-api-access-2ppdx\") pod \"opendatahub-operator-controller-manager-66d654f8fd-zb52s\" (UID: \"a384e39c-657d-4e8c-8c85-d41b60332868\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:13.902080 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.902015 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s"] Oct 30 22:06:13.942533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:13.942505 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:14.427325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:14.427283 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s"] Oct 30 22:06:14.435324 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:06:14.435285 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda384e39c_657d_4e8c_8c85_d41b60332868.slice/crio-5fa2f3767334ff2b349c8594e58b5d1addd47e8559901d752f9c17058df440ea WatchSource:0}: Error finding container 5fa2f3767334ff2b349c8594e58b5d1addd47e8559901d752f9c17058df440ea: Status 404 returned error can't find the container with id 5fa2f3767334ff2b349c8594e58b5d1addd47e8559901d752f9c17058df440ea Oct 30 22:06:15.003176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:15.003141 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" event={"ID":"a384e39c-657d-4e8c-8c85-d41b60332868","Type":"ContainerStarted","Data":"5fa2f3767334ff2b349c8594e58b5d1addd47e8559901d752f9c17058df440ea"} Oct 30 22:06:19.022054 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:19.022009 5226 generic.go:356] "Generic (PLEG): container finished" podID="a384e39c-657d-4e8c-8c85-d41b60332868" containerID="9d3c5c11a36ff410bbeb4fb4778fcdbad6045797fa1258345816e25010693862" exitCode=0 Oct 30 22:06:19.022486 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:19.022061 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" event={"ID":"a384e39c-657d-4e8c-8c85-d41b60332868","Type":"ContainerDied","Data":"9d3c5c11a36ff410bbeb4fb4778fcdbad6045797fa1258345816e25010693862"} Oct 30 22:06:21.033647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:21.033609 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" event={"ID":"a384e39c-657d-4e8c-8c85-d41b60332868","Type":"ContainerStarted","Data":"7f788d472043262453371a518117dea2d4f72cb0a03455156c36f820eb1e0235"} Oct 30 22:06:21.034048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:21.033728 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:21.310546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:21.310446 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" podStartSLOduration=3.235039955 podStartE2EDuration="8.310430673s" totalImagesPullingTime="4.576659688s" totalInitContainerRuntime="498.73103ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:21.263095153 +0000 UTC m=+272.520436927" watchObservedRunningTime="2026-10-30 22:06:21.310430673 +0000 UTC m=+272.567772383" Oct 30 22:06:24.409592 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:06:24.409554 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:06:30.205267 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.205168 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w"] Oct 30 22:06:30.207671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.207653 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" Oct 30 22:06:30.256893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.256858 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 22:06:30.303010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.302967 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 22:06:30.303166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.303023 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-zk865\"" Oct 30 22:06:30.304380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.304357 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w"] Oct 30 22:06:30.347563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.347534 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxzm9\" (UniqueName: \"kubernetes.io/projected/da8a1e8e-4f5c-4628-b0d3-08560730c7da-kube-api-access-gxzm9\") pod \"jobset-operator-674cd59dc5-sjw5w\" (UID: \"da8a1e8e-4f5c-4628-b0d3-08560730c7da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" Oct 30 22:06:30.347726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.347601 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8a1e8e-4f5c-4628-b0d3-08560730c7da-tmp\") pod \"jobset-operator-674cd59dc5-sjw5w\" (UID: \"da8a1e8e-4f5c-4628-b0d3-08560730c7da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" Oct 30 22:06:30.448962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.448928 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8a1e8e-4f5c-4628-b0d3-08560730c7da-tmp\") pod \"jobset-operator-674cd59dc5-sjw5w\" (UID: \"da8a1e8e-4f5c-4628-b0d3-08560730c7da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" Oct 30 22:06:30.449088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.448993 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxzm9\" (UniqueName: \"kubernetes.io/projected/da8a1e8e-4f5c-4628-b0d3-08560730c7da-kube-api-access-gxzm9\") pod \"jobset-operator-674cd59dc5-sjw5w\" (UID: \"da8a1e8e-4f5c-4628-b0d3-08560730c7da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" Oct 30 22:06:30.449372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.449353 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da8a1e8e-4f5c-4628-b0d3-08560730c7da-tmp\") pod \"jobset-operator-674cd59dc5-sjw5w\" (UID: \"da8a1e8e-4f5c-4628-b0d3-08560730c7da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" Oct 30 22:06:30.509835 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.509757 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxzm9\" (UniqueName: \"kubernetes.io/projected/da8a1e8e-4f5c-4628-b0d3-08560730c7da-kube-api-access-gxzm9\") pod \"jobset-operator-674cd59dc5-sjw5w\" (UID: \"da8a1e8e-4f5c-4628-b0d3-08560730c7da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" Oct 30 22:06:30.535687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.535658 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" Oct 30 22:06:30.913789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:30.913717 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w"] Oct 30 22:06:30.920745 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:06:30.920711 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda8a1e8e_4f5c_4628_b0d3_08560730c7da.slice/crio-edf0eb56a4134d4e89da85e0dafb2a0a858379f305f934de07a409aa5e5c946e WatchSource:0}: Error finding container edf0eb56a4134d4e89da85e0dafb2a0a858379f305f934de07a409aa5e5c946e: Status 404 returned error can't find the container with id edf0eb56a4134d4e89da85e0dafb2a0a858379f305f934de07a409aa5e5c946e Oct 30 22:06:31.074774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:31.074738 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" event={"ID":"da8a1e8e-4f5c-4628-b0d3-08560730c7da","Type":"ContainerStarted","Data":"edf0eb56a4134d4e89da85e0dafb2a0a858379f305f934de07a409aa5e5c946e"} Oct 30 22:06:32.039496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:32.039470 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66d654f8fd-zb52s" Oct 30 22:06:34.262930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.262886 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt"] Oct 30 22:06:34.266034 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.265997 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" Oct 30 22:06:34.284533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.284503 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3021c3f9-d44b-46f7-9295-effedab96873-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qxqnt\" (UID: \"3021c3f9-d44b-46f7-9295-effedab96873\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" Oct 30 22:06:34.284668 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.284583 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xspz6\" (UniqueName: \"kubernetes.io/projected/3021c3f9-d44b-46f7-9295-effedab96873-kube-api-access-xspz6\") pod \"cert-manager-cainjector-5c57bbd4fc-qxqnt\" (UID: \"3021c3f9-d44b-46f7-9295-effedab96873\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" Oct 30 22:06:34.323852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.323818 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt"] Oct 30 22:06:34.368382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.368333 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 22:06:34.368572 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.368413 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 22:06:34.368572 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.368482 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lkcnk\"" Oct 30 22:06:34.385730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.385639 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3021c3f9-d44b-46f7-9295-effedab96873-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qxqnt\" (UID: \"3021c3f9-d44b-46f7-9295-effedab96873\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" Oct 30 22:06:34.385730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.385728 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xspz6\" (UniqueName: \"kubernetes.io/projected/3021c3f9-d44b-46f7-9295-effedab96873-kube-api-access-xspz6\") pod \"cert-manager-cainjector-5c57bbd4fc-qxqnt\" (UID: \"3021c3f9-d44b-46f7-9295-effedab96873\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" Oct 30 22:06:34.444039 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.444003 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xspz6\" (UniqueName: \"kubernetes.io/projected/3021c3f9-d44b-46f7-9295-effedab96873-kube-api-access-xspz6\") pod \"cert-manager-cainjector-5c57bbd4fc-qxqnt\" (UID: \"3021c3f9-d44b-46f7-9295-effedab96873\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" Oct 30 22:06:34.444256 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.444230 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3021c3f9-d44b-46f7-9295-effedab96873-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qxqnt\" (UID: \"3021c3f9-d44b-46f7-9295-effedab96873\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" Oct 30 22:06:34.579224 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.579172 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" Oct 30 22:06:34.958670 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:34.958590 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt"] Oct 30 22:06:34.965544 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:06:34.965516 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3021c3f9_d44b_46f7_9295_effedab96873.slice/crio-0e2fdc9debb704022587b7515fd0bc4a96f00ab38ffd0c85167e55c77e443129 WatchSource:0}: Error finding container 0e2fdc9debb704022587b7515fd0bc4a96f00ab38ffd0c85167e55c77e443129: Status 404 returned error can't find the container with id 0e2fdc9debb704022587b7515fd0bc4a96f00ab38ffd0c85167e55c77e443129 Oct 30 22:06:35.096603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:35.096569 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" event={"ID":"3021c3f9-d44b-46f7-9295-effedab96873","Type":"ContainerStarted","Data":"0e2fdc9debb704022587b7515fd0bc4a96f00ab38ffd0c85167e55c77e443129"} Oct 30 22:06:35.097998 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:35.097967 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" event={"ID":"da8a1e8e-4f5c-4628-b0d3-08560730c7da","Type":"ContainerStarted","Data":"471abe364497920c39222866ac0195e78a2d4a79578c81fc6d6d87bfcad730a9"} Oct 30 22:06:35.213045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:35.212943 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sjw5w" podStartSLOduration=1.4595891779999999 podStartE2EDuration="5.21292536s" totalImagesPullingTime="3.753336182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:35.212253372 +0000 UTC m=+286.469595083" watchObservedRunningTime="2026-10-30 22:06:35.21292536 +0000 UTC m=+286.470267067" Oct 30 22:06:38.135176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.135140 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2"] Oct 30 22:06:38.142348 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.142312 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:38.219727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.219561 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3279a140-d7db-4643-8fbb-f41bf5057ff9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fcjp2\" (UID: \"3279a140-d7db-4643-8fbb-f41bf5057ff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:38.219727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.219701 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4zm\" (UniqueName: \"kubernetes.io/projected/3279a140-d7db-4643-8fbb-f41bf5057ff9-kube-api-access-2z4zm\") pod \"cert-manager-webhook-5dbbbf75cc-fcjp2\" (UID: \"3279a140-d7db-4643-8fbb-f41bf5057ff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:38.237936 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.237894 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zcd6d\"" Oct 30 22:06:38.239770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.239738 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2"] Oct 30 22:06:38.320974 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.320939 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4zm\" (UniqueName: \"kubernetes.io/projected/3279a140-d7db-4643-8fbb-f41bf5057ff9-kube-api-access-2z4zm\") pod \"cert-manager-webhook-5dbbbf75cc-fcjp2\" (UID: \"3279a140-d7db-4643-8fbb-f41bf5057ff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:38.321178 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.321012 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3279a140-d7db-4643-8fbb-f41bf5057ff9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fcjp2\" (UID: \"3279a140-d7db-4643-8fbb-f41bf5057ff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:38.379335 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.379301 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3279a140-d7db-4643-8fbb-f41bf5057ff9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-fcjp2\" (UID: \"3279a140-d7db-4643-8fbb-f41bf5057ff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:38.379929 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.379901 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4zm\" (UniqueName: \"kubernetes.io/projected/3279a140-d7db-4643-8fbb-f41bf5057ff9-kube-api-access-2z4zm\") pod \"cert-manager-webhook-5dbbbf75cc-fcjp2\" (UID: \"3279a140-d7db-4643-8fbb-f41bf5057ff9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:38.457717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.457690 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:38.851919 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:38.851836 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2"] Oct 30 22:06:38.859232 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:06:38.859187 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3279a140_d7db_4643_8fbb_f41bf5057ff9.slice/crio-8edcc25ce250fc022f90b0ae7f746ee390384cf85728875d3ce1b29a77d6a2b2 WatchSource:0}: Error finding container 8edcc25ce250fc022f90b0ae7f746ee390384cf85728875d3ce1b29a77d6a2b2: Status 404 returned error can't find the container with id 8edcc25ce250fc022f90b0ae7f746ee390384cf85728875d3ce1b29a77d6a2b2 Oct 30 22:06:39.118501 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:39.118407 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" event={"ID":"3279a140-d7db-4643-8fbb-f41bf5057ff9","Type":"ContainerStarted","Data":"8edcc25ce250fc022f90b0ae7f746ee390384cf85728875d3ce1b29a77d6a2b2"} Oct 30 22:06:40.126249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:40.126187 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" event={"ID":"3021c3f9-d44b-46f7-9295-effedab96873","Type":"ContainerStarted","Data":"5df02429585a7cb326afc5c7baec6b874a02094a975e95949f2c4f9c3d9d6d12"} Oct 30 22:06:40.237796 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:40.237732 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxqnt" podStartSLOduration=1.752430494 podStartE2EDuration="6.237716906s" totalImagesPullingTime="4.485286412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:40.236879969 +0000 UTC m=+291.494221679" watchObservedRunningTime="2026-10-30 22:06:40.237716906 +0000 UTC m=+291.495058615" Oct 30 22:06:41.132145 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:41.132106 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" event={"ID":"3279a140-d7db-4643-8fbb-f41bf5057ff9","Type":"ContainerStarted","Data":"bbea6c48017ee5d0dac47e43a2155a89c31f8b01964908d1308dd5d1ddd75e40"} Oct 30 22:06:41.132648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:41.132174 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:44.306907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.306847 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" podStartSLOduration=4.503650619 podStartE2EDuration="6.306832279s" totalImagesPullingTime="1.80318166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:41.249741144 +0000 UTC m=+292.507082853" watchObservedRunningTime="2026-10-30 22:06:44.306832279 +0000 UTC m=+295.564173988" Oct 30 22:06:44.307559 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.307540 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq"] Oct 30 22:06:44.310033 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.310009 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.374570 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.374540 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1df96253-235b-49f1-b70a-a87297020199-tmp\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.374727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.374704 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tdh\" (UniqueName: \"kubernetes.io/projected/1df96253-235b-49f1-b70a-a87297020199-kube-api-access-b8tdh\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.374794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.374779 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df96253-235b-49f1-b70a-a87297020199-tls-certs\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.407150 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.407119 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq"] Oct 30 22:06:44.454991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.454961 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 22:06:44.464654 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.464630 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 22:06:44.464654 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.464649 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ssrkh\"" Oct 30 22:06:44.476018 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.475998 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df96253-235b-49f1-b70a-a87297020199-tls-certs\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.476113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.476026 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1df96253-235b-49f1-b70a-a87297020199-tmp\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.476113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.476088 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tdh\" (UniqueName: \"kubernetes.io/projected/1df96253-235b-49f1-b70a-a87297020199-kube-api-access-b8tdh\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.478418 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.478399 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1df96253-235b-49f1-b70a-a87297020199-tmp\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.478670 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.478648 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df96253-235b-49f1-b70a-a87297020199-tls-certs\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.533272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.533252 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tdh\" (UniqueName: \"kubernetes.io/projected/1df96253-235b-49f1-b70a-a87297020199-kube-api-access-b8tdh\") pod \"kube-auth-proxy-784dfbd64d-xnvxq\" (UID: \"1df96253-235b-49f1-b70a-a87297020199\") " pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.621043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.620964 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" Oct 30 22:06:44.993033 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:44.992999 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq"] Oct 30 22:06:44.999142 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:06:44.999111 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df96253_235b_49f1_b70a_a87297020199.slice/crio-1036379c15fa63f82b71e26d572f6c820aa8b8772575235c7e373b5ebdb95cdf WatchSource:0}: Error finding container 1036379c15fa63f82b71e26d572f6c820aa8b8772575235c7e373b5ebdb95cdf: Status 404 returned error can't find the container with id 1036379c15fa63f82b71e26d572f6c820aa8b8772575235c7e373b5ebdb95cdf Oct 30 22:06:45.150512 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:45.150478 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" event={"ID":"1df96253-235b-49f1-b70a-a87297020199","Type":"ContainerStarted","Data":"1036379c15fa63f82b71e26d572f6c820aa8b8772575235c7e373b5ebdb95cdf"} Oct 30 22:06:47.139780 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:47.139748 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-fcjp2" Oct 30 22:06:48.930577 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:48.930553 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:06:48.930992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:48.930558 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:06:49.171319 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:49.171281 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" event={"ID":"1df96253-235b-49f1-b70a-a87297020199","Type":"ContainerStarted","Data":"58fed0f580d643f42bc6acfec47b2dc05de767ba42be2cc57ccbb479449ed5df"} Oct 30 22:06:49.284389 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:49.284295 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-784dfbd64d-xnvxq" podStartSLOduration=1.8991557700000001 podStartE2EDuration="5.284280204s" totalImagesPullingTime="3.385124434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:49.283124388 +0000 UTC m=+300.540466099" watchObservedRunningTime="2026-10-30 22:06:49.284280204 +0000 UTC m=+300.541621914" Oct 30 22:06:52.560140 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.560109 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-mr4qn"] Oct 30 22:06:52.638052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.638019 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mr4qn"] Oct 30 22:06:52.638221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.638120 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mr4qn" Oct 30 22:06:52.734950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.734916 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hwtzv\"" Oct 30 22:06:52.736242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.736222 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:06:52.748382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.748339 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de-bound-sa-token\") pod \"cert-manager-597b95cb84-mr4qn\" (UID: \"91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de\") " pod="cert-manager/cert-manager-597b95cb84-mr4qn" Oct 30 22:06:52.748382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.748381 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zww\" (UniqueName: \"kubernetes.io/projected/91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de-kube-api-access-b4zww\") pod \"cert-manager-597b95cb84-mr4qn\" (UID: \"91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de\") " pod="cert-manager/cert-manager-597b95cb84-mr4qn" Oct 30 22:06:52.751761 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.751742 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:52.783292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.783266 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:06:52.800860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.800808 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 22:06:52.800978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.800871 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 22:06:52.800978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.800920 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 22:06:52.801063 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.800996 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bcwpx\"" Oct 30 22:06:52.848959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.848893 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de-bound-sa-token\") pod \"cert-manager-597b95cb84-mr4qn\" (UID: \"91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de\") " pod="cert-manager/cert-manager-597b95cb84-mr4qn" Oct 30 22:06:52.848959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.848935 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zww\" (UniqueName: \"kubernetes.io/projected/91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de-kube-api-access-b4zww\") pod \"cert-manager-597b95cb84-mr4qn\" (UID: \"91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de\") " pod="cert-manager/cert-manager-597b95cb84-mr4qn" Oct 30 22:06:52.849100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.848978 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwbf\" (UniqueName: \"kubernetes.io/projected/b9bfbcb2-aa65-460a-bfa1-d8f30929d089-kube-api-access-skwbf\") pod \"limitador-limitador-7c9b87d574-frwkp\" (UID: \"b9bfbcb2-aa65-460a-bfa1-d8f30929d089\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:52.849100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.849026 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9bfbcb2-aa65-460a-bfa1-d8f30929d089-config-file\") pod \"limitador-limitador-7c9b87d574-frwkp\" (UID: \"b9bfbcb2-aa65-460a-bfa1-d8f30929d089\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:52.906974 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.906946 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zww\" (UniqueName: \"kubernetes.io/projected/91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de-kube-api-access-b4zww\") pod \"cert-manager-597b95cb84-mr4qn\" (UID: \"91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de\") " pod="cert-manager/cert-manager-597b95cb84-mr4qn" Oct 30 22:06:52.907107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.907020 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de-bound-sa-token\") pod \"cert-manager-597b95cb84-mr4qn\" (UID: \"91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de\") " pod="cert-manager/cert-manager-597b95cb84-mr4qn" Oct 30 22:06:52.950255 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.950199 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwbf\" (UniqueName: \"kubernetes.io/projected/b9bfbcb2-aa65-460a-bfa1-d8f30929d089-kube-api-access-skwbf\") pod \"limitador-limitador-7c9b87d574-frwkp\" (UID: \"b9bfbcb2-aa65-460a-bfa1-d8f30929d089\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:52.950395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.950289 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9bfbcb2-aa65-460a-bfa1-d8f30929d089-config-file\") pod \"limitador-limitador-7c9b87d574-frwkp\" (UID: \"b9bfbcb2-aa65-460a-bfa1-d8f30929d089\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:52.950891 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.950870 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9bfbcb2-aa65-460a-bfa1-d8f30929d089-config-file\") pod \"limitador-limitador-7c9b87d574-frwkp\" (UID: \"b9bfbcb2-aa65-460a-bfa1-d8f30929d089\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:52.967314 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:52.967294 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mr4qn" Oct 30 22:06:53.007519 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:53.007486 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwbf\" (UniqueName: \"kubernetes.io/projected/b9bfbcb2-aa65-460a-bfa1-d8f30929d089-kube-api-access-skwbf\") pod \"limitador-limitador-7c9b87d574-frwkp\" (UID: \"b9bfbcb2-aa65-460a-bfa1-d8f30929d089\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:53.012504 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:53.012476 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:06:53.082013 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:53.081980 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:53.348412 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:53.348373 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mr4qn"] Oct 30 22:06:53.354786 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:06:53.354763 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e68b2e_d5a9_45d2_a2c9_17b3bd5b52de.slice/crio-c2eb81c56895e1861d9074dec595442fb21f688c229db83b6dadf5d171444d31 WatchSource:0}: Error finding container c2eb81c56895e1861d9074dec595442fb21f688c229db83b6dadf5d171444d31: Status 404 returned error can't find the container with id c2eb81c56895e1861d9074dec595442fb21f688c229db83b6dadf5d171444d31 Oct 30 22:06:53.486390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:53.486359 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:06:53.492691 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:06:53.492657 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bfbcb2_aa65_460a_bfa1_d8f30929d089.slice/crio-d8fe6180891798665487efb9a72aa7d6be19bebe0f566ba98a62ffef223bf0ea WatchSource:0}: Error finding container d8fe6180891798665487efb9a72aa7d6be19bebe0f566ba98a62ffef223bf0ea: Status 404 returned error can't find the container with id d8fe6180891798665487efb9a72aa7d6be19bebe0f566ba98a62ffef223bf0ea Oct 30 22:06:53.494505 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:53.494489 5226 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:06:54.195080 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:54.195046 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mr4qn" event={"ID":"91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de","Type":"ContainerStarted","Data":"d73c5c13a978bcf054fefe05b4f9c160815d230b19283a01da1fefec8d7c449c"} Oct 30 22:06:54.195574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:54.195092 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mr4qn" event={"ID":"91e68b2e-d5a9-45d2-a2c9-17b3bd5b52de","Type":"ContainerStarted","Data":"c2eb81c56895e1861d9074dec595442fb21f688c229db83b6dadf5d171444d31"} Oct 30 22:06:54.196831 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:54.196801 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" event={"ID":"b9bfbcb2-aa65-460a-bfa1-d8f30929d089","Type":"ContainerStarted","Data":"d8fe6180891798665487efb9a72aa7d6be19bebe0f566ba98a62ffef223bf0ea"} Oct 30 22:06:54.303384 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:54.303323 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-mr4qn" podStartSLOduration=2.303306267 podStartE2EDuration="2.303306267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:54.303167433 +0000 UTC m=+305.560509160" watchObservedRunningTime="2026-10-30 22:06:54.303306267 +0000 UTC m=+305.560647978" Oct 30 22:06:54.409925 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:06:54.409889 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:06:57.213532 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:57.213493 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" event={"ID":"b9bfbcb2-aa65-460a-bfa1-d8f30929d089","Type":"ContainerStarted","Data":"63569faea443e64f4bb9812a7ef67d1a659b00cf3f7985becb73ae289c7e88b0"} Oct 30 22:06:57.213938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:57.213564 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:06:57.323196 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:06:57.323129 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" podStartSLOduration=1.980195294 podStartE2EDuration="5.323115696s" totalImagesPullingTime="3.342920402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:57.322514154 +0000 UTC m=+308.579855873" watchObservedRunningTime="2026-10-30 22:06:57.323115696 +0000 UTC m=+308.580457405" Oct 30 22:07:08.218604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:07:08.218574 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-frwkp" Oct 30 22:07:24.410070 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:07:24.410030 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:07:54.412361 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:07:54.412318 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:08:13.155930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.155850 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj"] Oct 30 22:08:13.159343 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.159324 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.241940 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.241909 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf913f4-f3ea-4908-b003-608ef8b6ecdc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.242097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.241947 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-maas-api-tls\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.242097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.241964 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psg94\" (UniqueName: \"kubernetes.io/projected/daf913f4-f3ea-4908-b003-608ef8b6ecdc-kube-api-access-psg94\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.242097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.242070 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-metrics-certs\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.255150 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.255126 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 22:08:13.255150 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.255126 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:08:13.255320 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.255155 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:13.255320 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.255148 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v8dzl\"" Oct 30 22:08:13.255320 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.255195 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:13.302394 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.302369 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj"] Oct 30 22:08:13.343028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.343000 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf913f4-f3ea-4908-b003-608ef8b6ecdc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.343163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.343038 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-maas-api-tls\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.343163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.343053 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psg94\" (UniqueName: \"kubernetes.io/projected/daf913f4-f3ea-4908-b003-608ef8b6ecdc-kube-api-access-psg94\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.343163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.343077 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-metrics-certs\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.343731 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.343708 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf913f4-f3ea-4908-b003-608ef8b6ecdc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.345606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.345581 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-metrics-certs\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.345606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.345601 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-maas-api-tls\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.400659 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.400633 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psg94\" (UniqueName: \"kubernetes.io/projected/daf913f4-f3ea-4908-b003-608ef8b6ecdc-kube-api-access-psg94\") pod \"maas-api-6d669c7fcd-kq6fj\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.471702 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.471679 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:13.835576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:13.835546 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj"] Oct 30 22:08:13.838235 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:08:13.838190 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf913f4_f3ea_4908_b003_608ef8b6ecdc.slice/crio-b9729e21f69a1db9fbc4273948dd2afc16d009079bc412160a20af9b5e8c9900 WatchSource:0}: Error finding container b9729e21f69a1db9fbc4273948dd2afc16d009079bc412160a20af9b5e8c9900: Status 404 returned error can't find the container with id b9729e21f69a1db9fbc4273948dd2afc16d009079bc412160a20af9b5e8c9900 Oct 30 22:08:14.550292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:14.550257 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" event={"ID":"daf913f4-f3ea-4908-b003-608ef8b6ecdc","Type":"ContainerStarted","Data":"b9729e21f69a1db9fbc4273948dd2afc16d009079bc412160a20af9b5e8c9900"} Oct 30 22:08:16.997162 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:16.997125 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mqnkq"] Oct 30 22:08:17.001286 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.001260 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:17.096210 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.096166 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 22:08:17.096391 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.096374 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-v5llf\"" Oct 30 22:08:17.096450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.096373 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 22:08:17.096450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.096415 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 22:08:17.097838 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.097819 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mqnkq"] Oct 30 22:08:17.148304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.148254 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442"] Oct 30 22:08:17.153453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.153424 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:17.172460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.172433 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442"] Oct 30 22:08:17.182400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.182372 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tt96\" (UniqueName: \"kubernetes.io/projected/a98769cf-c26c-41fd-8705-f0fac5e64e38-kube-api-access-5tt96\") pod \"kserve-controller-manager-776cc66d47-mqnkq\" (UID: \"a98769cf-c26c-41fd-8705-f0fac5e64e38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:17.182534 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.182515 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a98769cf-c26c-41fd-8705-f0fac5e64e38-cert\") pod \"kserve-controller-manager-776cc66d47-mqnkq\" (UID: \"a98769cf-c26c-41fd-8705-f0fac5e64e38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:17.214003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.213984 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 22:08:17.214107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.214088 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-twbvb\"" Oct 30 22:08:17.249287 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.246707 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tsw2f"] Oct 30 22:08:17.252194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.252173 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:17.283748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.283709 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a98769cf-c26c-41fd-8705-f0fac5e64e38-cert\") pod \"kserve-controller-manager-776cc66d47-mqnkq\" (UID: \"a98769cf-c26c-41fd-8705-f0fac5e64e38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:17.283878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.283774 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29dkl\" (UniqueName: \"kubernetes.io/projected/52a8d6e6-a3db-474e-ab06-9b63bedac21b-kube-api-access-29dkl\") pod \"llmisvc-controller-manager-869d5dcc5d-p4442\" (UID: \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:17.283878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.283852 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tt96\" (UniqueName: \"kubernetes.io/projected/a98769cf-c26c-41fd-8705-f0fac5e64e38-kube-api-access-5tt96\") pod \"kserve-controller-manager-776cc66d47-mqnkq\" (UID: \"a98769cf-c26c-41fd-8705-f0fac5e64e38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:17.283988 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.283923 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52a8d6e6-a3db-474e-ab06-9b63bedac21b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p4442\" (UID: \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:17.286610 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.286585 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a98769cf-c26c-41fd-8705-f0fac5e64e38-cert\") pod \"kserve-controller-manager-776cc66d47-mqnkq\" (UID: \"a98769cf-c26c-41fd-8705-f0fac5e64e38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:17.298217 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.298179 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tsw2f"] Oct 30 22:08:17.314008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.313982 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9p9vq\"" Oct 30 22:08:17.314113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.314054 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 22:08:17.349175 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.348769 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-tj2r4"] Oct 30 22:08:17.352568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.352517 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tt96\" (UniqueName: \"kubernetes.io/projected/a98769cf-c26c-41fd-8705-f0fac5e64e38-kube-api-access-5tt96\") pod \"kserve-controller-manager-776cc66d47-mqnkq\" (UID: \"a98769cf-c26c-41fd-8705-f0fac5e64e38\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:17.352679 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.352613 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.384906 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.384879 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75qm\" (UniqueName: \"kubernetes.io/projected/2b67e0dd-8f53-4ed2-9f23-960b95aa0711-kube-api-access-c75qm\") pod \"model-serving-api-75bd89fdfc-tsw2f\" (UID: \"2b67e0dd-8f53-4ed2-9f23-960b95aa0711\") " pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:17.385107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.384927 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b67e0dd-8f53-4ed2-9f23-960b95aa0711-tls-certs\") pod \"model-serving-api-75bd89fdfc-tsw2f\" (UID: \"2b67e0dd-8f53-4ed2-9f23-960b95aa0711\") " pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:17.385107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.384952 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29dkl\" (UniqueName: \"kubernetes.io/projected/52a8d6e6-a3db-474e-ab06-9b63bedac21b-kube-api-access-29dkl\") pod \"llmisvc-controller-manager-869d5dcc5d-p4442\" (UID: \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:17.385107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.385085 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52a8d6e6-a3db-474e-ab06-9b63bedac21b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p4442\" (UID: \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:17.387461 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.387431 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-tj2r4"] Oct 30 22:08:17.388111 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.388086 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52a8d6e6-a3db-474e-ab06-9b63bedac21b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p4442\" (UID: \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:17.414854 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.414826 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5rd2b\"" Oct 30 22:08:17.414854 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.414825 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 22:08:17.414854 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.414884 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 22:08:17.455244 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.455145 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29dkl\" (UniqueName: \"kubernetes.io/projected/52a8d6e6-a3db-474e-ab06-9b63bedac21b-kube-api-access-29dkl\") pod \"llmisvc-controller-manager-869d5dcc5d-p4442\" (UID: \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:17.466852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.466821 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:17.486387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.486360 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75qm\" (UniqueName: \"kubernetes.io/projected/2b67e0dd-8f53-4ed2-9f23-960b95aa0711-kube-api-access-c75qm\") pod \"model-serving-api-75bd89fdfc-tsw2f\" (UID: \"2b67e0dd-8f53-4ed2-9f23-960b95aa0711\") " pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:17.486503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.486407 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4064f6b8-6293-4953-a7a1-260e0ec8945b-metrics-certs\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.486503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.486448 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b67e0dd-8f53-4ed2-9f23-960b95aa0711-tls-certs\") pod \"model-serving-api-75bd89fdfc-tsw2f\" (UID: \"2b67e0dd-8f53-4ed2-9f23-960b95aa0711\") " pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:17.486503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.486472 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4064f6b8-6293-4953-a7a1-260e0ec8945b-cert\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.486678 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.486564 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfv7d\" (UniqueName: \"kubernetes.io/projected/4064f6b8-6293-4953-a7a1-260e0ec8945b-kube-api-access-dfv7d\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.488814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.488788 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b67e0dd-8f53-4ed2-9f23-960b95aa0711-tls-certs\") pod \"model-serving-api-75bd89fdfc-tsw2f\" (UID: \"2b67e0dd-8f53-4ed2-9f23-960b95aa0711\") " pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:17.558790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.558750 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75qm\" (UniqueName: \"kubernetes.io/projected/2b67e0dd-8f53-4ed2-9f23-960b95aa0711-kube-api-access-c75qm\") pod \"model-serving-api-75bd89fdfc-tsw2f\" (UID: \"2b67e0dd-8f53-4ed2-9f23-960b95aa0711\") " pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:17.567580 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.567353 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:17.569614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.569585 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" event={"ID":"daf913f4-f3ea-4908-b003-608ef8b6ecdc","Type":"ContainerStarted","Data":"dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de"} Oct 30 22:08:17.569856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.569806 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:17.587948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.587921 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4064f6b8-6293-4953-a7a1-260e0ec8945b-cert\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.588042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.588012 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfv7d\" (UniqueName: \"kubernetes.io/projected/4064f6b8-6293-4953-a7a1-260e0ec8945b-kube-api-access-dfv7d\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.588134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.588119 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4064f6b8-6293-4953-a7a1-260e0ec8945b-metrics-certs\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.591090 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.591063 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4064f6b8-6293-4953-a7a1-260e0ec8945b-cert\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.591182 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.591066 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4064f6b8-6293-4953-a7a1-260e0ec8945b-metrics-certs\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.616193 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.615961 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:17.651325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.651292 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfv7d\" (UniqueName: \"kubernetes.io/projected/4064f6b8-6293-4953-a7a1-260e0ec8945b-kube-api-access-dfv7d\") pod \"odh-model-controller-769875db56-tj2r4\" (UID: \"4064f6b8-6293-4953-a7a1-260e0ec8945b\") " pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.671546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.671504 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:17.688837 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.688287 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" podStartSLOduration=1.366278167 podStartE2EDuration="4.688272434s" totalImagesPullingTime="3.321994267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:17.68662693 +0000 UTC m=+388.943968680" watchObservedRunningTime="2026-10-30 22:08:17.688272434 +0000 UTC m=+388.945614144" Oct 30 22:08:17.851152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.851120 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442"] Oct 30 22:08:17.854896 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:08:17.854868 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod52a8d6e6_a3db_474e_ab06_9b63bedac21b.slice/crio-40a54677a1d7a316f45794621a8ceb50fc8fe08f2b59592376f5ff49e825ba91 WatchSource:0}: Error finding container 40a54677a1d7a316f45794621a8ceb50fc8fe08f2b59592376f5ff49e825ba91: Status 404 returned error can't find the container with id 40a54677a1d7a316f45794621a8ceb50fc8fe08f2b59592376f5ff49e825ba91 Oct 30 22:08:17.950397 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:17.950346 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tsw2f"] Oct 30 22:08:17.953683 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:08:17.953642 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b67e0dd_8f53_4ed2_9f23_960b95aa0711.slice/crio-36ab5e514783636076d7c32de441d139bafbd3e6080517d361c0134b2e80382e WatchSource:0}: Error finding container 36ab5e514783636076d7c32de441d139bafbd3e6080517d361c0134b2e80382e: Status 404 returned error can't find the container with id 36ab5e514783636076d7c32de441d139bafbd3e6080517d361c0134b2e80382e Oct 30 22:08:18.053614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:18.052981 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:08:18.053614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:18.053093 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mqnkq"] Oct 30 22:08:18.053614 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:08:18.053558 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98769cf_c26c_41fd_8705_f0fac5e64e38.slice/crio-b4717ea55d79ee4bb798a0c9e09f72bdd7a98a1ee4c2b56e59919882aee2fcf2 WatchSource:0}: Error finding container b4717ea55d79ee4bb798a0c9e09f72bdd7a98a1ee4c2b56e59919882aee2fcf2: Status 404 returned error can't find the container with id b4717ea55d79ee4bb798a0c9e09f72bdd7a98a1ee4c2b56e59919882aee2fcf2 Oct 30 22:08:18.068185 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:18.068164 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-tj2r4"] Oct 30 22:08:18.070684 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:08:18.070656 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4064f6b8_6293_4953_a7a1_260e0ec8945b.slice/crio-b4e511cb52fc282e05a3d5f275ad0997e43c5f9c77e2a648af8aa5546addc701 WatchSource:0}: Error finding container b4e511cb52fc282e05a3d5f275ad0997e43c5f9c77e2a648af8aa5546addc701: Status 404 returned error can't find the container with id b4e511cb52fc282e05a3d5f275ad0997e43c5f9c77e2a648af8aa5546addc701 Oct 30 22:08:18.575689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:18.575650 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-tj2r4" event={"ID":"4064f6b8-6293-4953-a7a1-260e0ec8945b","Type":"ContainerStarted","Data":"b4e511cb52fc282e05a3d5f275ad0997e43c5f9c77e2a648af8aa5546addc701"} Oct 30 22:08:18.576763 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:18.576739 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" event={"ID":"52a8d6e6-a3db-474e-ab06-9b63bedac21b","Type":"ContainerStarted","Data":"40a54677a1d7a316f45794621a8ceb50fc8fe08f2b59592376f5ff49e825ba91"} Oct 30 22:08:18.577815 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:18.577790 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" event={"ID":"2b67e0dd-8f53-4ed2-9f23-960b95aa0711","Type":"ContainerStarted","Data":"36ab5e514783636076d7c32de441d139bafbd3e6080517d361c0134b2e80382e"} Oct 30 22:08:18.578812 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:18.578791 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" event={"ID":"a98769cf-c26c-41fd-8705-f0fac5e64e38","Type":"ContainerStarted","Data":"b4717ea55d79ee4bb798a0c9e09f72bdd7a98a1ee4c2b56e59919882aee2fcf2"} Oct 30 22:08:22.620743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.620559 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" event={"ID":"a98769cf-c26c-41fd-8705-f0fac5e64e38","Type":"ContainerStarted","Data":"aa29327146911b9c0043086830b9e071e25121bdd9252f45d392715767ca9102"} Oct 30 22:08:22.620743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.620644 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:22.622925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.622897 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" event={"ID":"52a8d6e6-a3db-474e-ab06-9b63bedac21b","Type":"ContainerStarted","Data":"121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67"} Oct 30 22:08:22.623077 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.623059 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:22.625113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.625064 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" event={"ID":"2b67e0dd-8f53-4ed2-9f23-960b95aa0711","Type":"ContainerStarted","Data":"55a2413d022b1640652448948ee8e255dc7348f9f8280d848634cc735d98a093"} Oct 30 22:08:22.625410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.625355 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:22.730647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.730588 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" podStartSLOduration=2.308977594 podStartE2EDuration="6.730571571s" totalImagesPullingTime="4.421593977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:22.729581288 +0000 UTC m=+393.986923026" watchObservedRunningTime="2026-10-30 22:08:22.730571571 +0000 UTC m=+393.987913280" Oct 30 22:08:22.838921 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.838855 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" podStartSLOduration=1.375705572 podStartE2EDuration="5.838839529s" totalImagesPullingTime="4.463133957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:22.838304778 +0000 UTC m=+394.095646487" watchObservedRunningTime="2026-10-30 22:08:22.838839529 +0000 UTC m=+394.096181249" Oct 30 22:08:22.950038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:22.947959 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" podStartSLOduration=1.879692174 podStartE2EDuration="5.947941271s" totalImagesPullingTime="4.068249097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:22.947720467 +0000 UTC m=+394.205062197" watchObservedRunningTime="2026-10-30 22:08:22.947941271 +0000 UTC m=+394.205282980" Oct 30 22:08:23.585176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:23.585141 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:23.632466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:23.632424 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-tj2r4" event={"ID":"4064f6b8-6293-4953-a7a1-260e0ec8945b","Type":"ContainerStarted","Data":"7e93ebf95c2223d4bb04471d4a8a9dd4d6765219f07506ea2a041c6041e40dc7"} Oct 30 22:08:23.632902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:23.632885 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:23.804803 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:23.804753 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-tj2r4" podStartSLOduration=1.805772509 podStartE2EDuration="6.804738974s" totalImagesPullingTime="4.998966465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:23.803730002 +0000 UTC m=+395.061071712" watchObservedRunningTime="2026-10-30 22:08:23.804738974 +0000 UTC m=+395.062080729" Oct 30 22:08:24.412003 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:08:24.411956 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:08:25.643965 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:25.643931 5226 generic.go:356] "Generic (PLEG): container finished" podID="3a989905-c9be-4423-8d8c-83400f966607" containerID="55558ea82e0caff68e3ca06b526f67413f279e8fd002004e31d77e2ec2c62aff" exitCode=0 Oct 30 22:08:25.644473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:25.644013 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" event={"ID":"3a989905-c9be-4423-8d8c-83400f966607","Type":"ContainerDied","Data":"55558ea82e0caff68e3ca06b526f67413f279e8fd002004e31d77e2ec2c62aff"} Oct 30 22:08:25.644473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:25.644425 5226 scope.go:122] "RemoveContainer" containerID="55558ea82e0caff68e3ca06b526f67413f279e8fd002004e31d77e2ec2c62aff" Oct 30 22:08:25.645609 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:25.645578 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:08:25.645692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:25.645652 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:08:25.645692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:25.645683 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:08:26.656589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:26.656556 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" event={"ID":"3a989905-c9be-4423-8d8c-83400f966607","Type":"ContainerStarted","Data":"b386f522932713a12b2ea2cf15c562fb1366e02bb91bc692278261df3442c8e2"} Oct 30 22:08:27.540229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:27.535620 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:08:27.540229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:27.535760 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nmwcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body= Oct 30 22:08:27.540229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:27.535807 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" Oct 30 22:08:27.842393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:27.842313 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:08:28.539636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:28.539596 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nmwcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body= Oct 30 22:08:28.539898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:28.539874 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" Oct 30 22:08:29.546229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:29.538557 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nmwcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body= Oct 30 22:08:29.546229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:29.538617 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" Oct 30 22:08:30.535226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:30.534102 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nmwcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body= Oct 30 22:08:30.535226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:30.534170 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" Oct 30 22:08:31.534606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:31.534566 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nmwcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body= Oct 30 22:08:31.535063 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:31.534623 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" Oct 30 22:08:32.538790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:32.538759 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:08:32.539857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:32.539837 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:08:33.637992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:33.637964 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-tsw2f" Oct 30 22:08:34.317828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.317795 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj"] Oct 30 22:08:34.318022 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.318000 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" podUID="daf913f4-f3ea-4908-b003-608ef8b6ecdc" containerName="maas-api" containerID="cri-o://dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de" gracePeriod=30 Oct 30 22:08:34.587551 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.587522 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:34.641552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.641523 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-tj2r4" Oct 30 22:08:34.693511 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.693473 5226 generic.go:356] "Generic (PLEG): container finished" podID="daf913f4-f3ea-4908-b003-608ef8b6ecdc" containerID="dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de" exitCode=0 Oct 30 22:08:34.693671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.693517 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" Oct 30 22:08:34.693671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.693562 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" event={"ID":"daf913f4-f3ea-4908-b003-608ef8b6ecdc","Type":"ContainerDied","Data":"dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de"} Oct 30 22:08:34.693671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.693618 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj" event={"ID":"daf913f4-f3ea-4908-b003-608ef8b6ecdc","Type":"ContainerDied","Data":"b9729e21f69a1db9fbc4273948dd2afc16d009079bc412160a20af9b5e8c9900"} Oct 30 22:08:34.693671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.693642 5226 scope.go:122] "RemoveContainer" containerID="dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de" Oct 30 22:08:34.703983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.703380 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf913f4-f3ea-4908-b003-608ef8b6ecdc-kube-api-access-psg94\" (UniqueName: \"kubernetes.io/projected/daf913f4-f3ea-4908-b003-608ef8b6ecdc-kube-api-access-psg94\") pod \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " Oct 30 22:08:34.703983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.703431 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf913f4-f3ea-4908-b003-608ef8b6ecdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf913f4-f3ea-4908-b003-608ef8b6ecdc-openshift-service-ca\") pod \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " Oct 30 22:08:34.703983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.703486 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-maas-api-tls\") pod \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " Oct 30 22:08:34.703983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.703537 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-metrics-certs\") pod \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\" (UID: \"daf913f4-f3ea-4908-b003-608ef8b6ecdc\") " Oct 30 22:08:34.703983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.703834 5226 scope.go:122] "RemoveContainer" containerID="dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de" Oct 30 22:08:34.704490 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:08:34.704465 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de\": container with ID starting with dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de not found: ID does not exist" containerID="dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de" Oct 30 22:08:34.704579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.704515 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de"} err="failed to get container status \"dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de\": rpc error: code = NotFound desc = could not find container \"dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de\": container with ID starting with dba0f6f88076cdcaf544b3e7c43099ca9f4b79d39e2d089086bb2111ff93a7de not found: ID does not exist" Oct 30 22:08:34.704805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.704773 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf913f4-f3ea-4908-b003-608ef8b6ecdc-openshift-service-ca" pod "daf913f4-f3ea-4908-b003-608ef8b6ecdc" (UID: "daf913f4-f3ea-4908-b003-608ef8b6ecdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:34.706492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.706461 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-metrics-certs" pod "daf913f4-f3ea-4908-b003-608ef8b6ecdc" (UID: "daf913f4-f3ea-4908-b003-608ef8b6ecdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:34.706582 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.706498 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-maas-api-tls" pod "daf913f4-f3ea-4908-b003-608ef8b6ecdc" (UID: "daf913f4-f3ea-4908-b003-608ef8b6ecdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:34.706817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.706798 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf913f4-f3ea-4908-b003-608ef8b6ecdc-kube-api-access-psg94" pod "daf913f4-f3ea-4908-b003-608ef8b6ecdc" (UID: "daf913f4-f3ea-4908-b003-608ef8b6ecdc"). InnerVolumeSpecName "kube-api-access-psg94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:34.804555 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.804514 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psg94\" (UniqueName: \"kubernetes.io/projected/daf913f4-f3ea-4908-b003-608ef8b6ecdc-kube-api-access-psg94\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.804555 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.804550 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daf913f4-f3ea-4908-b003-608ef8b6ecdc-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.804555 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.804560 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:34.804789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:34.804571 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daf913f4-f3ea-4908-b003-608ef8b6ecdc-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:35.100293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:35.100214 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj"] Oct 30 22:08:35.104736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:35.104708 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-kq6fj"] Oct 30 22:08:36.904056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:36.904021 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf913f4-f3ea-4908-b003-608ef8b6ecdc" path="/var/lib/kubelet/pods/daf913f4-f3ea-4908-b003-608ef8b6ecdc/volumes" Oct 30 22:08:53.634788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:53.634748 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:08:53.638440 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:08:53.638417 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mqnkq" Oct 30 22:08:54.410068 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:08:54.410033 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:09:24.410911 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:09:24.410870 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:09:29.377017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.376937 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2"] Oct 30 22:09:29.377441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.377345 5226 cpu_manager.go:414] "containerMap: removing container" podUID="daf913f4-f3ea-4908-b003-608ef8b6ecdc" containerName="maas-api" Oct 30 22:09:29.377441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.377357 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf913f4-f3ea-4908-b003-608ef8b6ecdc" containerName="maas-api" Oct 30 22:09:29.377441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.377414 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf913f4-f3ea-4908-b003-608ef8b6ecdc" containerName="maas-api" Oct 30 22:09:29.379522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.379505 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:29.436508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.436469 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2"] Oct 30 22:09:29.578818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.578775 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntbq\" (UniqueName: \"kubernetes.io/projected/10fa6181-ca6f-4767-8373-7338944442b9-kube-api-access-tntbq\") pod \"llmisvc-controller-manager-6466c554d8-s2ws2\" (UID: \"10fa6181-ca6f-4767-8373-7338944442b9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:29.579009 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.578923 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10fa6181-ca6f-4767-8373-7338944442b9-cert\") pod \"llmisvc-controller-manager-6466c554d8-s2ws2\" (UID: \"10fa6181-ca6f-4767-8373-7338944442b9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:29.679768 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.679729 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10fa6181-ca6f-4767-8373-7338944442b9-cert\") pod \"llmisvc-controller-manager-6466c554d8-s2ws2\" (UID: \"10fa6181-ca6f-4767-8373-7338944442b9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:29.679948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.679826 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntbq\" (UniqueName: \"kubernetes.io/projected/10fa6181-ca6f-4767-8373-7338944442b9-kube-api-access-tntbq\") pod \"llmisvc-controller-manager-6466c554d8-s2ws2\" (UID: \"10fa6181-ca6f-4767-8373-7338944442b9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:29.682106 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.682084 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10fa6181-ca6f-4767-8373-7338944442b9-cert\") pod \"llmisvc-controller-manager-6466c554d8-s2ws2\" (UID: \"10fa6181-ca6f-4767-8373-7338944442b9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:29.738151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.738114 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntbq\" (UniqueName: \"kubernetes.io/projected/10fa6181-ca6f-4767-8373-7338944442b9-kube-api-access-tntbq\") pod \"llmisvc-controller-manager-6466c554d8-s2ws2\" (UID: \"10fa6181-ca6f-4767-8373-7338944442b9\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:29.991330 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:29.991249 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:30.354310 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:30.354282 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2"] Oct 30 22:09:30.357095 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:09:30.356748 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fa6181_ca6f_4767_8373_7338944442b9.slice/crio-c5d76e242a349e4915efbe11ec6d027fe04e9bddfe2fa6280d9655f20824d0f6 WatchSource:0}: Error finding container c5d76e242a349e4915efbe11ec6d027fe04e9bddfe2fa6280d9655f20824d0f6: Status 404 returned error can't find the container with id c5d76e242a349e4915efbe11ec6d027fe04e9bddfe2fa6280d9655f20824d0f6 Oct 30 22:09:30.985417 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:30.985382 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" event={"ID":"10fa6181-ca6f-4767-8373-7338944442b9","Type":"ContainerStarted","Data":"c5d76e242a349e4915efbe11ec6d027fe04e9bddfe2fa6280d9655f20824d0f6"} Oct 30 22:09:31.992011 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:31.991979 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" event={"ID":"10fa6181-ca6f-4767-8373-7338944442b9","Type":"ContainerStarted","Data":"0f7468e2d23cf8e64b5ba0ff21f1f90a523c45cb9d74c97bac4facabd53d5ee6"} Oct 30 22:09:31.992477 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:31.992044 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:09:32.102645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:09:32.102590 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" podStartSLOduration=2.299542729 podStartE2EDuration="3.102575754s" totalImagesPullingTime="803.033025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:32.101490766 +0000 UTC m=+463.358832477" watchObservedRunningTime="2026-10-30 22:09:32.102575754 +0000 UTC m=+463.359917464" Oct 30 22:09:54.410167 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:09:54.410123 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:10:02.999560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:02.999526 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s2ws2" Oct 30 22:10:03.157619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.157579 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442"] Oct 30 22:10:03.157858 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.157828 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" podUID="52a8d6e6-a3db-474e-ab06-9b63bedac21b" containerName="manager" containerID="cri-o://121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67" gracePeriod=30 Oct 30 22:10:03.402569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.402545 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:10:03.467725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.467693 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a8d6e6-a3db-474e-ab06-9b63bedac21b-kube-api-access-29dkl\" (UniqueName: \"kubernetes.io/projected/52a8d6e6-a3db-474e-ab06-9b63bedac21b-kube-api-access-29dkl\") pod \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\" (UID: \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\") " Oct 30 22:10:03.467904 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.467785 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a8d6e6-a3db-474e-ab06-9b63bedac21b-cert\" (UniqueName: \"kubernetes.io/secret/52a8d6e6-a3db-474e-ab06-9b63bedac21b-cert\") pod \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\" (UID: \"52a8d6e6-a3db-474e-ab06-9b63bedac21b\") " Oct 30 22:10:03.469962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.469921 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a8d6e6-a3db-474e-ab06-9b63bedac21b-cert" pod "52a8d6e6-a3db-474e-ab06-9b63bedac21b" (UID: "52a8d6e6-a3db-474e-ab06-9b63bedac21b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:03.470082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.469981 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a8d6e6-a3db-474e-ab06-9b63bedac21b-kube-api-access-29dkl" pod "52a8d6e6-a3db-474e-ab06-9b63bedac21b" (UID: "52a8d6e6-a3db-474e-ab06-9b63bedac21b"). InnerVolumeSpecName "kube-api-access-29dkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:03.569061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.568973 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29dkl\" (UniqueName: \"kubernetes.io/projected/52a8d6e6-a3db-474e-ab06-9b63bedac21b-kube-api-access-29dkl\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:03.569061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:03.569003 5226 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52a8d6e6-a3db-474e-ab06-9b63bedac21b-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.135751 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.135718 5226 generic.go:356] "Generic (PLEG): container finished" podID="52a8d6e6-a3db-474e-ab06-9b63bedac21b" containerID="121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67" exitCode=0 Oct 30 22:10:04.135751 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.135755 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" event={"ID":"52a8d6e6-a3db-474e-ab06-9b63bedac21b","Type":"ContainerDied","Data":"121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67"} Oct 30 22:10:04.136237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.135764 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" Oct 30 22:10:04.136237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.135793 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442" event={"ID":"52a8d6e6-a3db-474e-ab06-9b63bedac21b","Type":"ContainerDied","Data":"40a54677a1d7a316f45794621a8ceb50fc8fe08f2b59592376f5ff49e825ba91"} Oct 30 22:10:04.136237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.135814 5226 scope.go:122] "RemoveContainer" containerID="121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67" Oct 30 22:10:04.145017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.144997 5226 scope.go:122] "RemoveContainer" containerID="121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67" Oct 30 22:10:04.145331 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:10:04.145303 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67\": container with ID starting with 121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67 not found: ID does not exist" containerID="121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67" Oct 30 22:10:04.145405 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.145341 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67"} err="failed to get container status \"121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67\": rpc error: code = NotFound desc = could not find container \"121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67\": container with ID starting with 121615d04c22bda239549cf30fbabedd021211a7285d142ee4a6f9cdae0b7f67 not found: ID does not exist" Oct 30 22:10:04.288807 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.288776 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442"] Oct 30 22:10:04.334492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.334461 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p4442"] Oct 30 22:10:04.903524 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:04.903491 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a8d6e6-a3db-474e-ab06-9b63bedac21b" path="/var/lib/kubelet/pods/52a8d6e6-a3db-474e-ab06-9b63bedac21b/volumes" Oct 30 22:10:24.409580 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:10:24.409544 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:10:26.324387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:26.324354 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:10:40.906996 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:40.906958 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:10:54.409994 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:10:54.409953 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:10:58.637061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:10:58.637029 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:11:24.410170 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:11:24.410129 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:11:48.970557 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:11:48.970529 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:11:48.970557 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:11:48.970559 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:11:54.411033 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:11:54.410996 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:12:24.410700 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:12:24.410658 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:12:31.972947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:31.972806 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j"] Oct 30 22:12:31.973470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:31.973234 5226 cpu_manager.go:414] "containerMap: removing container" podUID="52a8d6e6-a3db-474e-ab06-9b63bedac21b" containerName="manager" Oct 30 22:12:31.973470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:31.973251 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a8d6e6-a3db-474e-ab06-9b63bedac21b" containerName="manager" Oct 30 22:12:31.973470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:31.973322 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a8d6e6-a3db-474e-ab06-9b63bedac21b" containerName="manager" Oct 30 22:12:31.975478 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:31.975463 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:32.072752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.072715 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-23f3d4-dockercfg-sdhjx\"" Oct 30 22:12:32.072752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.072715 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-23f3d4\"" Oct 30 22:12:32.074230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.074197 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j"] Oct 30 22:12:32.081117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.081095 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f42v\" (UniqueName: \"kubernetes.io/projected/3575f09c-1832-468b-b938-1fd125084dce-kube-api-access-8f42v\") pod \"payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j\" (UID: \"3575f09c-1832-468b-b938-1fd125084dce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:32.081195 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.081169 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3575f09c-1832-468b-b938-1fd125084dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j\" (UID: \"3575f09c-1832-468b-b938-1fd125084dce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:32.120305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.120274 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf"] Oct 30 22:12:32.123027 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.123009 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:32.169894 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.169850 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf"] Oct 30 22:12:32.181821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.181785 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f42v\" (UniqueName: \"kubernetes.io/projected/3575f09c-1832-468b-b938-1fd125084dce-kube-api-access-8f42v\") pod \"payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j\" (UID: \"3575f09c-1832-468b-b938-1fd125084dce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:32.181987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.181863 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3575f09c-1832-468b-b938-1fd125084dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j\" (UID: \"3575f09c-1832-468b-b938-1fd125084dce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:32.182575 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.182551 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3575f09c-1832-468b-b938-1fd125084dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j\" (UID: \"3575f09c-1832-468b-b938-1fd125084dce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:32.243632 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.243554 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f42v\" (UniqueName: \"kubernetes.io/projected/3575f09c-1832-468b-b938-1fd125084dce-kube-api-access-8f42v\") pod \"payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j\" (UID: \"3575f09c-1832-468b-b938-1fd125084dce\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:32.282642 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.282606 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf\" (UID: \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:32.282794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.282716 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fr8z\" (UniqueName: \"kubernetes.io/projected/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-kube-api-access-7fr8z\") pod \"payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf\" (UID: \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:32.287527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.287506 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:32.383945 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.383914 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf\" (UID: \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:32.384107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.383985 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fr8z\" (UniqueName: \"kubernetes.io/projected/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-kube-api-access-7fr8z\") pod \"payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf\" (UID: \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:32.384726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.384704 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf\" (UID: \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:32.443680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.443647 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fr8z\" (UniqueName: \"kubernetes.io/projected/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-kube-api-access-7fr8z\") pod \"payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf\" (UID: \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:32.650512 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.650409 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j"] Oct 30 22:12:32.653174 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:12:32.653133 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3575f09c_1832_468b_b938_1fd125084dce.slice/crio-78cae9b7b8a1fc61cbc8ae7b05b637102222a0849e9180339709f76006ff38a1 WatchSource:0}: Error finding container 78cae9b7b8a1fc61cbc8ae7b05b637102222a0849e9180339709f76006ff38a1: Status 404 returned error can't find the container with id 78cae9b7b8a1fc61cbc8ae7b05b637102222a0849e9180339709f76006ff38a1 Oct 30 22:12:32.655143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.655122 5226 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:12:32.734140 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.734109 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:32.840431 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:32.840401 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" event={"ID":"3575f09c-1832-468b-b938-1fd125084dce","Type":"ContainerStarted","Data":"78cae9b7b8a1fc61cbc8ae7b05b637102222a0849e9180339709f76006ff38a1"} Oct 30 22:12:33.103805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:33.103774 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf"] Oct 30 22:12:33.106919 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:12:33.106891 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e743f3_5463_4ae1_9c8a_75a892fbaceb.slice/crio-91054eda83fe3391586464d2c8c47a68eac756d20fb5cc7b2af9c70f57ea9b76 WatchSource:0}: Error finding container 91054eda83fe3391586464d2c8c47a68eac756d20fb5cc7b2af9c70f57ea9b76: Status 404 returned error can't find the container with id 91054eda83fe3391586464d2c8c47a68eac756d20fb5cc7b2af9c70f57ea9b76 Oct 30 22:12:33.845851 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:33.845808 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" event={"ID":"a7e743f3-5463-4ae1-9c8a-75a892fbaceb","Type":"ContainerStarted","Data":"91054eda83fe3391586464d2c8c47a68eac756d20fb5cc7b2af9c70f57ea9b76"} Oct 30 22:12:34.631179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:34.631145 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:12:35.858766 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:35.858728 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" event={"ID":"3575f09c-1832-468b-b938-1fd125084dce","Type":"ContainerStarted","Data":"eae287d37ea1a123f1dc742c407176339be3debe822ef3a7339e8a0caefa9fac"} Oct 30 22:12:35.859240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:35.858796 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:35.968375 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:35.968250 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" podStartSLOduration=1.933468331 podStartE2EDuration="4.96822952s" totalImagesPullingTime="3.034761189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:35.967598442 +0000 UTC m=+647.224940154" watchObservedRunningTime="2026-10-30 22:12:35.96822952 +0000 UTC m=+647.225571232" Oct 30 22:12:36.865596 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:36.865549 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" event={"ID":"a7e743f3-5463-4ae1-9c8a-75a892fbaceb","Type":"ContainerStarted","Data":"59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137"} Oct 30 22:12:36.866040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:36.865782 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:36.974918 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:36.974850 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" podStartSLOduration=1.982484354 podStartE2EDuration="4.97483312s" totalImagesPullingTime="2.992348766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:36.973990405 +0000 UTC m=+648.231332112" watchObservedRunningTime="2026-10-30 22:12:36.97483312 +0000 UTC m=+648.232174831" Oct 30 22:12:39.142877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.142845 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8"] Oct 30 22:12:39.146096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.146080 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:39.194770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.194745 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-be17e7\"" Oct 30 22:12:39.240698 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.240667 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-be17e7-dockercfg-2cnlq\"" Oct 30 22:12:39.243316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.243290 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8"] Oct 30 22:12:39.251064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.251037 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kbmb\" (UniqueName: \"kubernetes.io/projected/80a86fe5-a6d3-4b72-869c-e983f3d693d9-kube-api-access-4kbmb\") pod \"payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8\" (UID: \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:39.251192 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.251074 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a86fe5-a6d3-4b72-869c-e983f3d693d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8\" (UID: \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:39.351877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.351841 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kbmb\" (UniqueName: \"kubernetes.io/projected/80a86fe5-a6d3-4b72-869c-e983f3d693d9-kube-api-access-4kbmb\") pod \"payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8\" (UID: \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:39.352050 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.351886 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a86fe5-a6d3-4b72-869c-e983f3d693d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8\" (UID: \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:39.352570 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.352551 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a86fe5-a6d3-4b72-869c-e983f3d693d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8\" (UID: \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:39.410225 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.410130 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kbmb\" (UniqueName: \"kubernetes.io/projected/80a86fe5-a6d3-4b72-869c-e983f3d693d9-kube-api-access-4kbmb\") pod \"payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8\" (UID: \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:39.458286 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.458248 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:39.830797 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.830766 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8"] Oct 30 22:12:39.834042 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:12:39.834013 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a86fe5_a6d3_4b72_869c_e983f3d693d9.slice/crio-ff88ad30caf8fda38e9f3600e1826a169775e7f9f32de78c1eff9a95590cb91a WatchSource:0}: Error finding container ff88ad30caf8fda38e9f3600e1826a169775e7f9f32de78c1eff9a95590cb91a: Status 404 returned error can't find the container with id ff88ad30caf8fda38e9f3600e1826a169775e7f9f32de78c1eff9a95590cb91a Oct 30 22:12:39.881067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:39.881040 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" event={"ID":"80a86fe5-a6d3-4b72-869c-e983f3d693d9","Type":"ContainerStarted","Data":"ff88ad30caf8fda38e9f3600e1826a169775e7f9f32de78c1eff9a95590cb91a"} Oct 30 22:12:40.888991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:40.888954 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" event={"ID":"80a86fe5-a6d3-4b72-869c-e983f3d693d9","Type":"ContainerStarted","Data":"e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5"} Oct 30 22:12:40.889448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:40.889066 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:41.001403 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.001345 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" podStartSLOduration=2.001328797 podStartE2EDuration="2.001328797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:41.000129019 +0000 UTC m=+652.257470749" watchObservedRunningTime="2026-10-30 22:12:41.001328797 +0000 UTC m=+652.258670546" Oct 30 22:12:41.695272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.695236 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx"] Oct 30 22:12:41.698071 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.698052 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.793346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.793311 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:41.793346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.793311 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 22:12:41.793558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.793363 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:41.793558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.793314 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-f80a0e-serving-cert\"" Oct 30 22:12:41.793558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.793383 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v8dzl\"" Oct 30 22:12:41.795024 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.795006 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx"] Oct 30 22:12:41.874045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.874016 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.874221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.874082 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-metrics-certs\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.874221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.874162 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9nhm\" (UniqueName: \"kubernetes.io/projected/808ad9a0-6994-4616-8b45-e28b1ef83e4e-kube-api-access-f9nhm\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.874221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.874199 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808ad9a0-6994-4616-8b45-e28b1ef83e4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.975673 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.975587 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808ad9a0-6994-4616-8b45-e28b1ef83e4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.975673 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.975658 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.976111 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.975738 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-metrics-certs\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.976111 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.975775 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9nhm\" (UniqueName: \"kubernetes.io/projected/808ad9a0-6994-4616-8b45-e28b1ef83e4e-kube-api-access-f9nhm\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.976111 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:12:41.975838 5226 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-serving-cert: secret "maas-api-e2e-worker-w5-f80a0e-serving-cert" not found Oct 30 22:12:41.976111 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:12:41.975934 5226 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls podName:808ad9a0-6994-4616-8b45-e28b1ef83e4e nodeName:}" failed. No retries permitted until 2026-10-30 22:12:42.475910454 +0000 UTC m=+653.733252161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls") pod "maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" (UID: "808ad9a0-6994-4616-8b45-e28b1ef83e4e") : secret "maas-api-e2e-worker-w5-f80a0e-serving-cert" not found Oct 30 22:12:41.976361 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.976343 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808ad9a0-6994-4616-8b45-e28b1ef83e4e-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:41.978169 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:41.978145 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-metrics-certs\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:42.033576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:42.033546 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9nhm\" (UniqueName: \"kubernetes.io/projected/808ad9a0-6994-4616-8b45-e28b1ef83e4e-kube-api-access-f9nhm\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:42.478979 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:42.478941 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:42.481462 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:42.481426 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls\") pod \"maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:42.609843 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:42.609807 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:42.976382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:42.976351 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx"] Oct 30 22:12:42.978913 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:12:42.978888 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808ad9a0_6994_4616_8b45_e28b1ef83e4e.slice/crio-ef0552dc513348e5ce4e097f40374123a732c35f1c251849e28c62ba257cfa5e WatchSource:0}: Error finding container ef0552dc513348e5ce4e097f40374123a732c35f1c251849e28c62ba257cfa5e: Status 404 returned error can't find the container with id ef0552dc513348e5ce4e097f40374123a732c35f1c251849e28c62ba257cfa5e Oct 30 22:12:43.917736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:43.917705 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" event={"ID":"808ad9a0-6994-4616-8b45-e28b1ef83e4e","Type":"ContainerStarted","Data":"ef0552dc513348e5ce4e097f40374123a732c35f1c251849e28c62ba257cfa5e"} Oct 30 22:12:43.925898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:43.925862 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:12:44.924736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:44.924698 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" event={"ID":"808ad9a0-6994-4616-8b45-e28b1ef83e4e","Type":"ContainerStarted","Data":"045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383"} Oct 30 22:12:44.925159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:44.924837 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:45.036828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:45.036767 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" podStartSLOduration=3.127166309 podStartE2EDuration="4.036751437s" totalImagesPullingTime="909.585128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:45.035941968 +0000 UTC m=+656.293283678" watchObservedRunningTime="2026-10-30 22:12:45.036751437 +0000 UTC m=+656.294093147" Oct 30 22:12:46.308245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.308194 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb"] Oct 30 22:12:46.311879 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.311858 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:46.360968 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.360935 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d2d4c3\"" Oct 30 22:12:46.406496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.406460 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d2d4c3-dockercfg-w74km\"" Oct 30 22:12:46.408141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.408115 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb"] Oct 30 22:12:46.417121 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.417082 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkbs\" (UniqueName: \"kubernetes.io/projected/7376dc43-4a28-47ef-8905-9fdb0100c2dc-kube-api-access-8nkbs\") pod \"payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb\" (UID: \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:46.417310 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.417247 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376dc43-4a28-47ef-8905-9fdb0100c2dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb\" (UID: \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:46.518324 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.518286 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376dc43-4a28-47ef-8905-9fdb0100c2dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb\" (UID: \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:46.518495 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.518441 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkbs\" (UniqueName: \"kubernetes.io/projected/7376dc43-4a28-47ef-8905-9fdb0100c2dc-kube-api-access-8nkbs\") pod \"payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb\" (UID: \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:46.518955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.518936 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376dc43-4a28-47ef-8905-9fdb0100c2dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb\" (UID: \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:46.576895 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.576810 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkbs\" (UniqueName: \"kubernetes.io/projected/7376dc43-4a28-47ef-8905-9fdb0100c2dc-kube-api-access-8nkbs\") pod \"payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb\" (UID: \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:46.624947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.624905 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:46.869338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.869261 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:12:46.992086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:46.992053 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb"] Oct 30 22:12:46.996261 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:12:46.996233 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7376dc43_4a28_47ef_8905_9fdb0100c2dc.slice/crio-be96b580a87d2dce02ac11b95accbfc46df7759ccfdd28ead8eb2c28a749af95 WatchSource:0}: Error finding container be96b580a87d2dce02ac11b95accbfc46df7759ccfdd28ead8eb2c28a749af95: Status 404 returned error can't find the container with id be96b580a87d2dce02ac11b95accbfc46df7759ccfdd28ead8eb2c28a749af95 Oct 30 22:12:47.874030 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:47.873993 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:12:47.946424 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:47.946389 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" event={"ID":"7376dc43-4a28-47ef-8905-9fdb0100c2dc","Type":"ContainerStarted","Data":"839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf"} Oct 30 22:12:47.946591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:47.946441 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:47.946591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:47.946459 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" event={"ID":"7376dc43-4a28-47ef-8905-9fdb0100c2dc","Type":"ContainerStarted","Data":"be96b580a87d2dce02ac11b95accbfc46df7759ccfdd28ead8eb2c28a749af95"} Oct 30 22:12:48.092033 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:48.091975 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" podStartSLOduration=2.091959625 podStartE2EDuration="2.091959625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:48.090758537 +0000 UTC m=+659.348100238" watchObservedRunningTime="2026-10-30 22:12:48.091959625 +0000 UTC m=+659.349301335" Oct 30 22:12:50.941611 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:50.941578 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:12:51.897124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:51.897095 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:12:54.410761 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:12:54.410725 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:12:56.329663 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.329627 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:12:56.534246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.534200 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv"] Oct 30 22:12:56.541216 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.541177 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.597845 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.597762 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.597845 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.597818 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.598037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.597879 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgtq\" (UniqueName: \"kubernetes.io/projected/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-kube-api-access-9pgtq\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.598037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.597938 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-metrics-certs\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.636017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.635982 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b13755-serving-cert\"" Oct 30 22:12:56.637679 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.637656 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv"] Oct 30 22:12:56.699151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.699117 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.699355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.699155 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgtq\" (UniqueName: \"kubernetes.io/projected/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-kube-api-access-9pgtq\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.699355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.699225 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-metrics-certs\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.699355 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:12:56.699306 5226 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-serving-cert: secret "maas-api-e2e-worker-w4-b13755-serving-cert" not found Oct 30 22:12:56.699355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.699344 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.699531 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:12:56.699419 5226 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls podName:e1cfb0fc-ba6f-4133-94db-01f8c7fed971 nodeName:}" failed. No retries permitted until 2026-10-30 22:12:57.19939839 +0000 UTC m=+668.456740095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls") pod "maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" (UID: "e1cfb0fc-ba6f-4133-94db-01f8c7fed971") : secret "maas-api-e2e-worker-w4-b13755-serving-cert" not found Oct 30 22:12:56.699935 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.699915 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.701593 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.701574 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-metrics-certs\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:56.757437 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:56.757395 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgtq\" (UniqueName: \"kubernetes.io/projected/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-kube-api-access-9pgtq\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:57.207148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:57.207106 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:57.209717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:57.209685 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:57.454640 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:57.454597 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:57.823644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:57.823326 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv"] Oct 30 22:12:57.826059 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:12:57.826030 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cfb0fc_ba6f_4133_94db_01f8c7fed971.slice/crio-0183884a89817a0fc1dc1a261792c66ad5ca5cf9dc825a3d6da8aa9488e10ac4 WatchSource:0}: Error finding container 0183884a89817a0fc1dc1a261792c66ad5ca5cf9dc825a3d6da8aa9488e10ac4: Status 404 returned error can't find the container with id 0183884a89817a0fc1dc1a261792c66ad5ca5cf9dc825a3d6da8aa9488e10ac4 Oct 30 22:12:57.995357 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:57.995319 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" event={"ID":"e1cfb0fc-ba6f-4133-94db-01f8c7fed971","Type":"ContainerStarted","Data":"0183884a89817a0fc1dc1a261792c66ad5ca5cf9dc825a3d6da8aa9488e10ac4"} Oct 30 22:12:58.955266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:58.955230 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:12:59.001504 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:59.001424 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" event={"ID":"e1cfb0fc-ba6f-4133-94db-01f8c7fed971","Type":"ContainerStarted","Data":"87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc"} Oct 30 22:12:59.001504 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:59.001495 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:12:59.175789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:12:59.175732 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" podStartSLOduration=2.294526534 podStartE2EDuration="3.175715357s" totalImagesPullingTime="881.188823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:59.17465555 +0000 UTC m=+670.431997271" watchObservedRunningTime="2026-10-30 22:12:59.175715357 +0000 UTC m=+670.433057067" Oct 30 22:13:05.010921 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:05.010883 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:13:06.458888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:06.458858 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:13:15.081511 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.081481 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7"] Oct 30 22:13:15.085275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.085252 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.166424 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.166393 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-home\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.166424 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.166429 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-dshm\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.166614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.166446 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-model-cache\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.166614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.166591 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tls-certs\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.166682 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.166625 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmrs\" (UniqueName: \"kubernetes.io/projected/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-kube-api-access-8fmrs\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.166749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.166724 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tmp-dir\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.180392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.180366 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"simulator-w1-af30a9-kserve-self-signed-certs\"" Oct 30 22:13:15.180520 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.180399 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"default-dockercfg-kv4tv\"" Oct 30 22:13:15.180520 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.180452 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"openshift-service-ca.crt\"" Oct 30 22:13:15.180637 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.180598 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"kube-root-ca.crt\"" Oct 30 22:13:15.182423 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.182402 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7"] Oct 30 22:13:15.267589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.267555 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-home\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.267589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.267591 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-dshm\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.267829 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.267610 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-model-cache\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.267829 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.267648 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tls-certs\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.267829 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.267669 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmrs\" (UniqueName: \"kubernetes.io/projected/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-kube-api-access-8fmrs\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.267829 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.267703 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tmp-dir\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.268062 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.268034 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-home\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.268122 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.268069 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-model-cache\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.268162 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.268121 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tmp-dir\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.270002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.269984 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-dshm\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.270114 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.270098 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tls-certs\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.325521 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.325491 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmrs\" (UniqueName: \"kubernetes.io/projected/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-kube-api-access-8fmrs\") pod \"simulator-w1-af30a9-kserve-5885f48f86-jllm7\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.397106 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.397034 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:15.766285 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:15.766261 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7"] Oct 30 22:13:15.768552 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:13:15.768523 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02c9879f_87e6_4a5b_af84_5fdaaf41ae54.slice/crio-d1fa1ed6551189bf1a5f134eec676a76c2c6336a67e5b5e6e3c40a7f825ee5a3 WatchSource:0}: Error finding container d1fa1ed6551189bf1a5f134eec676a76c2c6336a67e5b5e6e3c40a7f825ee5a3: Status 404 returned error can't find the container with id d1fa1ed6551189bf1a5f134eec676a76c2c6336a67e5b5e6e3c40a7f825ee5a3 Oct 30 22:13:16.084484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:16.084400 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" event={"ID":"02c9879f-87e6-4a5b-af84-5fdaaf41ae54","Type":"ContainerStarted","Data":"d1fa1ed6551189bf1a5f134eec676a76c2c6336a67e5b5e6e3c40a7f825ee5a3"} Oct 30 22:13:18.096602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:18.096563 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" event={"ID":"02c9879f-87e6-4a5b-af84-5fdaaf41ae54","Type":"ContainerStarted","Data":"6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e"} Oct 30 22:13:18.207186 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:18.207130 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podStartSLOduration=1.646686196 podStartE2EDuration="3.20711091s" totalImagesPullingTime="1.560424714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:18.205467958 +0000 UTC m=+689.462809668" watchObservedRunningTime="2026-10-30 22:13:18.20711091 +0000 UTC m=+689.464452622" Oct 30 22:13:18.397919 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:18.397842 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:22.041394 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:22.041363 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:13:24.410637 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:13:24.410601 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:13:25.397276 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:25.397242 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:25.410168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:25.410146 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:26.144658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:26.144633 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:13:47.101912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.101876 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w"] Oct 30 22:13:47.105582 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.105563 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.167809 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.167750 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgns\" (UniqueName: \"kubernetes.io/projected/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-kube-api-access-5fgns\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.167977 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.167921 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-model-cache\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.168029 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.167976 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tmp-dir\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.168091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.168055 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-dshm\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.168149 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.168087 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-home\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.168149 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.168109 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tls-certs\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.200061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.200034 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"simulator-w3-23f3d4-kserve-self-signed-certs\"" Oct 30 22:13:47.200194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.200086 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"openshift-service-ca.crt\"" Oct 30 22:13:47.200194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.200106 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"default-dockercfg-6kxxw\"" Oct 30 22:13:47.200194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.200143 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"kube-root-ca.crt\"" Oct 30 22:13:47.202083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.202067 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w"] Oct 30 22:13:47.268839 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.268807 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-model-cache\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.269004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.268853 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tmp-dir\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.269004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.268872 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-dshm\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.269004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.268891 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-home\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.269155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.269005 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tls-certs\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.269155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.269060 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgns\" (UniqueName: \"kubernetes.io/projected/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-kube-api-access-5fgns\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.269322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.269301 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tmp-dir\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.269456 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.269315 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-model-cache\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.269456 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.269365 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-home\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.271178 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.271158 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-dshm\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.271513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.271495 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tls-certs\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.326979 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.326946 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgns\" (UniqueName: \"kubernetes.io/projected/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-kube-api-access-5fgns\") pod \"simulator-w3-23f3d4-kserve-666c764954-vwh6w\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.419896 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.419802 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:47.805236 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:47.805193 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w"] Oct 30 22:13:47.807687 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:13:47.807659 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b5c60fa_3c6b_408c_84f5_b4d070aa8fb0.slice/crio-002ce3164b005f140e5a29f86196a80ca10ad6fc5d73b4ddc41bf24dda081faf WatchSource:0}: Error finding container 002ce3164b005f140e5a29f86196a80ca10ad6fc5d73b4ddc41bf24dda081faf: Status 404 returned error can't find the container with id 002ce3164b005f140e5a29f86196a80ca10ad6fc5d73b4ddc41bf24dda081faf Oct 30 22:13:48.247371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:48.247338 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" event={"ID":"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0","Type":"ContainerStarted","Data":"6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b"} Oct 30 22:13:48.247764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:48.247380 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" event={"ID":"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0","Type":"ContainerStarted","Data":"002ce3164b005f140e5a29f86196a80ca10ad6fc5d73b4ddc41bf24dda081faf"} Oct 30 22:13:48.361475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:48.361422 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podStartSLOduration=1.361409696 podStartE2EDuration="1.361409696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:48.359871768 +0000 UTC m=+719.617213457" watchObservedRunningTime="2026-10-30 22:13:48.361409696 +0000 UTC m=+719.618751405" Oct 30 22:13:48.420224 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:48.420177 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:51.176821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.176788 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c"] Oct 30 22:13:51.183515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.183490 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.227185 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.227156 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-model-cache\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.227185 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.227188 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d3c91d-e41a-4a37-8367-53540a9f98b2-tls-certs\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.227377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.227234 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-dshm\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.227377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.227341 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s82v\" (UniqueName: \"kubernetes.io/projected/38d3c91d-e41a-4a37-8367-53540a9f98b2-kube-api-access-9s82v\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.227377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.227372 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-tmp-dir\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.227543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.227519 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-home\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.281254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.281217 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"openshift-service-ca.crt\"" Oct 30 22:13:51.281254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.281228 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"kube-root-ca.crt\"" Oct 30 22:13:51.281445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.281260 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"default-dockercfg-4bjl8\"" Oct 30 22:13:51.281445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.281282 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"simulator-w2-eb4aa1-kserve-self-signed-certs\"" Oct 30 22:13:51.286236 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.286219 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c"] Oct 30 22:13:51.328054 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328028 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-dshm\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.328222 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328068 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s82v\" (UniqueName: \"kubernetes.io/projected/38d3c91d-e41a-4a37-8367-53540a9f98b2-kube-api-access-9s82v\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.328222 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328093 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-tmp-dir\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.328222 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328163 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-home\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.328404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328247 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-model-cache\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.328404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328270 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d3c91d-e41a-4a37-8367-53540a9f98b2-tls-certs\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.328591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328564 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-tmp-dir\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.328651 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328596 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-home\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.328701 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.328646 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-model-cache\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.330401 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.330373 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-dshm\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.330638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.330622 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d3c91d-e41a-4a37-8367-53540a9f98b2-tls-certs\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.387032 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.387008 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s82v\" (UniqueName: \"kubernetes.io/projected/38d3c91d-e41a-4a37-8367-53540a9f98b2-kube-api-access-9s82v\") pod \"simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.496221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.496109 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:51.876974 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:51.876950 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c"] Oct 30 22:13:51.878835 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:13:51.878807 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38d3c91d_e41a_4a37_8367_53540a9f98b2.slice/crio-da779fae194887775902792169630b7a35cf7edfc92e03196c5f5a5e33efa842 WatchSource:0}: Error finding container da779fae194887775902792169630b7a35cf7edfc92e03196c5f5a5e33efa842: Status 404 returned error can't find the container with id da779fae194887775902792169630b7a35cf7edfc92e03196c5f5a5e33efa842 Oct 30 22:13:52.269122 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:52.269083 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" event={"ID":"38d3c91d-e41a-4a37-8367-53540a9f98b2","Type":"ContainerStarted","Data":"27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c"} Oct 30 22:13:52.269541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:52.269132 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" event={"ID":"38d3c91d-e41a-4a37-8367-53540a9f98b2","Type":"ContainerStarted","Data":"da779fae194887775902792169630b7a35cf7edfc92e03196c5f5a5e33efa842"} Oct 30 22:13:52.383281 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:52.383229 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podStartSLOduration=1.383198494 podStartE2EDuration="1.383198494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:52.382331815 +0000 UTC m=+723.639673524" watchObservedRunningTime="2026-10-30 22:13:52.383198494 +0000 UTC m=+723.640540203" Oct 30 22:13:52.496956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:52.496904 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:13:54.410056 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:13:54.410020 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:13:54.955588 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:54.955553 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:13:57.420340 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:57.420305 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:57.432786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:57.432752 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:13:58.308271 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:13:58.308245 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:14:01.497169 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:01.497088 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:14:01.511872 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:01.511845 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:14:02.328915 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:02.328889 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:14:04.522488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.522444 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl"] Oct 30 22:14:04.526197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.526169 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.621715 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.621678 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"openshift-service-ca.crt\"" Oct 30 22:14:04.621885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.621733 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"kube-root-ca.crt\"" Oct 30 22:14:04.621885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.621802 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"simulator-w5-f80a0e-kserve-self-signed-certs\"" Oct 30 22:14:04.621885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.621859 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"default-dockercfg-b9tzf\"" Oct 30 22:14:04.623857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.623835 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl"] Oct 30 22:14:04.658949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.658917 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-home\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.659112 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.658951 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-dshm\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.659112 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.659075 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-tmp-dir\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.659112 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.659099 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-model-cache\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.659263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.659125 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcwls\" (UniqueName: \"kubernetes.io/projected/7458e0f5-fb73-4a8f-9554-40f302510a37-kube-api-access-bcwls\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.659263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.659145 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7458e0f5-fb73-4a8f-9554-40f302510a37-tls-certs\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760341 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-home\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760384 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-dshm\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760448 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-tmp-dir\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760470 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-model-cache\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760488 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcwls\" (UniqueName: \"kubernetes.io/projected/7458e0f5-fb73-4a8f-9554-40f302510a37-kube-api-access-bcwls\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760512 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7458e0f5-fb73-4a8f-9554-40f302510a37-tls-certs\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760883 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-home\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760912 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-tmp-dir\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.760983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.760959 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-model-cache\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.762778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.762755 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-dshm\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.763283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.763260 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7458e0f5-fb73-4a8f-9554-40f302510a37-tls-certs\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.819352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.819275 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcwls\" (UniqueName: \"kubernetes.io/projected/7458e0f5-fb73-4a8f-9554-40f302510a37-kube-api-access-bcwls\") pod \"simulator-w5-f80a0e-kserve-5655fffb58-qqmbl\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:04.838598 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:04.838568 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:05.212107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:05.212081 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl"] Oct 30 22:14:05.214220 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:14:05.214182 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7458e0f5_fb73_4a8f_9554_40f302510a37.slice/crio-2ef7b5dca02c6f8e8b990176f89719e3e95d1097f7c1a85bf58e9b887a8d48b7 WatchSource:0}: Error finding container 2ef7b5dca02c6f8e8b990176f89719e3e95d1097f7c1a85bf58e9b887a8d48b7: Status 404 returned error can't find the container with id 2ef7b5dca02c6f8e8b990176f89719e3e95d1097f7c1a85bf58e9b887a8d48b7 Oct 30 22:14:05.335574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:05.335535 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" event={"ID":"7458e0f5-fb73-4a8f-9554-40f302510a37","Type":"ContainerStarted","Data":"efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485"} Oct 30 22:14:05.335574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:05.335579 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" event={"ID":"7458e0f5-fb73-4a8f-9554-40f302510a37","Type":"ContainerStarted","Data":"2ef7b5dca02c6f8e8b990176f89719e3e95d1097f7c1a85bf58e9b887a8d48b7"} Oct 30 22:14:05.445265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:05.445176 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podStartSLOduration=1.445158809 podStartE2EDuration="1.445158809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:05.444706543 +0000 UTC m=+736.702048280" watchObservedRunningTime="2026-10-30 22:14:05.445158809 +0000 UTC m=+736.702500521" Oct 30 22:14:05.838794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:05.838764 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:14.839026 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:14.838991 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:14.851418 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:14.851385 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:15.395472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:15.395442 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:14:21.476684 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:21.476646 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:14:22.018199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.018166 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp"] Oct 30 22:14:22.022875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.022853 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.116955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.116921 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-af30a9\"/\"premium-w1-af30a9-kserve-self-signed-certs\"" Oct 30 22:14:22.118909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.118883 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp"] Oct 30 22:14:22.133596 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.133570 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-dshm\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.133726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.133618 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b5c2\" (UniqueName: \"kubernetes.io/projected/b741662d-6efd-40f1-88f9-4fc558cd61c6-kube-api-access-9b5c2\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.133781 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.133748 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-model-cache\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.133834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.133782 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-tmp-dir\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.133834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.133806 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-home\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.133930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.133902 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b741662d-6efd-40f1-88f9-4fc558cd61c6-tls-certs\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.234842 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.234805 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-model-cache\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.234842 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.234849 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-tmp-dir\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.235087 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.234872 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-home\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.235087 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.235058 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b741662d-6efd-40f1-88f9-4fc558cd61c6-tls-certs\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.235194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.235137 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-dshm\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.235266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.235226 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b5c2\" (UniqueName: \"kubernetes.io/projected/b741662d-6efd-40f1-88f9-4fc558cd61c6-kube-api-access-9b5c2\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.235325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.235265 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-home\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.235325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.235303 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-tmp-dir\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.235411 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.235333 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-model-cache\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.237403 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.237376 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-dshm\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.237700 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.237680 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b741662d-6efd-40f1-88f9-4fc558cd61c6-tls-certs\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.292985 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.292905 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b5c2\" (UniqueName: \"kubernetes.io/projected/b741662d-6efd-40f1-88f9-4fc558cd61c6-kube-api-access-9b5c2\") pod \"premium-w1-af30a9-kserve-5cb777ddbc-2h2lp\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.334755 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.334716 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:22.719506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:22.719479 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp"] Oct 30 22:14:22.721318 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:14:22.721291 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb741662d_6efd_40f1_88f9_4fc558cd61c6.slice/crio-dac04d469b2417c1c673a50eaccfd9bf42986804a11357e77887e14f67de62f7 WatchSource:0}: Error finding container dac04d469b2417c1c673a50eaccfd9bf42986804a11357e77887e14f67de62f7: Status 404 returned error can't find the container with id dac04d469b2417c1c673a50eaccfd9bf42986804a11357e77887e14f67de62f7 Oct 30 22:14:23.436191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:23.436148 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" event={"ID":"b741662d-6efd-40f1-88f9-4fc558cd61c6","Type":"ContainerStarted","Data":"abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a"} Oct 30 22:14:23.436381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:23.436200 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" event={"ID":"b741662d-6efd-40f1-88f9-4fc558cd61c6","Type":"ContainerStarted","Data":"dac04d469b2417c1c673a50eaccfd9bf42986804a11357e77887e14f67de62f7"} Oct 30 22:14:23.547915 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:23.547855 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podStartSLOduration=2.547842196 podStartE2EDuration="2.547842196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:23.547159477 +0000 UTC m=+754.804501187" watchObservedRunningTime="2026-10-30 22:14:23.547842196 +0000 UTC m=+754.805183906" Oct 30 22:14:24.335756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:24.335721 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:24.410072 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:14:24.410031 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:14:26.011163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.011129 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5"] Oct 30 22:14:26.014983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.014961 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.083453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.083420 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-home\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.083631 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.083488 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtg47\" (UniqueName: \"kubernetes.io/projected/15178a87-7ac5-405e-9b46-1a8de8861922-kube-api-access-jtg47\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.083631 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.083589 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-tmp-dir\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.083735 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.083650 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-model-cache\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.083844 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.083818 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-dshm\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.083923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.083881 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15178a87-7ac5-405e-9b46-1a8de8861922-tls-certs\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.110565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.110522 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"openshift-service-ca.crt\"" Oct 30 22:14:26.110565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.110550 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"kube-root-ca.crt\"" Oct 30 22:14:26.110758 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.110583 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"simulator-w4-b13755-kserve-self-signed-certs\"" Oct 30 22:14:26.110758 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.110673 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b13755\"/\"default-dockercfg-2zk59\"" Oct 30 22:14:26.120722 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.120692 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5"] Oct 30 22:14:26.185136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185102 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-model-cache\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.185319 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185179 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-dshm\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.185319 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185228 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15178a87-7ac5-405e-9b46-1a8de8861922-tls-certs\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.185319 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185267 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-home\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.185319 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185314 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtg47\" (UniqueName: \"kubernetes.io/projected/15178a87-7ac5-405e-9b46-1a8de8861922-kube-api-access-jtg47\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.185538 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185446 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-tmp-dir\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.185640 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185620 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-model-cache\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.185760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185738 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-home\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.185814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.185801 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-tmp-dir\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.187544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.187524 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-dshm\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.187747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.187729 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15178a87-7ac5-405e-9b46-1a8de8861922-tls-certs\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.243600 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.243568 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtg47\" (UniqueName: \"kubernetes.io/projected/15178a87-7ac5-405e-9b46-1a8de8861922-kube-api-access-jtg47\") pod \"simulator-w4-b13755-kserve-5c558499c6-s29m5\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.327141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.327056 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:26.698665 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:26.698632 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5"] Oct 30 22:14:26.702470 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:14:26.702440 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15178a87_7ac5_405e_9b46_1a8de8861922.slice/crio-9902d914543f1b13b56f84d83efbe25d1f25f98d4948d1ad50f8550d6a66a902 WatchSource:0}: Error finding container 9902d914543f1b13b56f84d83efbe25d1f25f98d4948d1ad50f8550d6a66a902: Status 404 returned error can't find the container with id 9902d914543f1b13b56f84d83efbe25d1f25f98d4948d1ad50f8550d6a66a902 Oct 30 22:14:27.460320 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:27.460288 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" event={"ID":"15178a87-7ac5-405e-9b46-1a8de8861922","Type":"ContainerStarted","Data":"9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39"} Oct 30 22:14:27.460998 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:27.460327 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" event={"ID":"15178a87-7ac5-405e-9b46-1a8de8861922","Type":"ContainerStarted","Data":"9902d914543f1b13b56f84d83efbe25d1f25f98d4948d1ad50f8550d6a66a902"} Oct 30 22:14:27.570406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:27.570354 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podStartSLOduration=2.570340084 podStartE2EDuration="2.570340084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:27.569231837 +0000 UTC m=+758.826573545" watchObservedRunningTime="2026-10-30 22:14:27.570340084 +0000 UTC m=+758.827681794" Oct 30 22:14:28.327565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:28.327532 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:32.334935 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:32.334892 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:32.347759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:32.347728 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:32.497675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:32.497648 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:14:36.328169 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:36.328142 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:36.340721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:36.340695 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:36.520008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:36.519980 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:14:47.044618 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:14:47.044576 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:14:54.410736 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:14:54.410695 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:15:00.186322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.186289 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw"] Oct 30 22:15:00.191479 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.191456 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:00.208853 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.208830 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws"] Oct 30 22:15:00.212284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.212267 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:00.215044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.215026 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk"] Oct 30 22:15:00.218475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.218370 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:00.224727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.224699 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a325e37-9e3c-4828-8a41-f88ed2a4acda-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-4pxcw\" (UID: \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:00.224814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.224753 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vhp\" (UniqueName: \"kubernetes.io/projected/9a325e37-9e3c-4828-8a41-f88ed2a4acda-kube-api-access-g4vhp\") pod \"maas-api-key-cleanup-29889975-4pxcw\" (UID: \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:00.253434 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.253412 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\"" Oct 30 22:15:00.279902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.279877 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw"] Oct 30 22:15:00.325833 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.325800 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws\" (UID: \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:00.326002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.325875 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b959\" (UniqueName: \"kubernetes.io/projected/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-kube-api-access-4b959\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws\" (UID: \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:00.326073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.326000 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a325e37-9e3c-4828-8a41-f88ed2a4acda-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-4pxcw\" (UID: \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:00.326073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.326053 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfct\" (UniqueName: \"kubernetes.io/projected/0b16d139-d87a-44b3-9e53-1f2764e561c7-kube-api-access-mpfct\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk\" (UID: \"0b16d139-d87a-44b3-9e53-1f2764e561c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:00.326191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.326082 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vhp\" (UniqueName: \"kubernetes.io/projected/9a325e37-9e3c-4828-8a41-f88ed2a4acda-kube-api-access-g4vhp\") pod \"maas-api-key-cleanup-29889975-4pxcw\" (UID: \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:00.326191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.326131 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b16d139-d87a-44b3-9e53-1f2764e561c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk\" (UID: \"0b16d139-d87a-44b3-9e53-1f2764e561c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:00.326944 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.326923 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a325e37-9e3c-4828-8a41-f88ed2a4acda-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-4pxcw\" (UID: \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:00.331250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.331182 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws"] Oct 30 22:15:00.333470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.333443 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk"] Oct 30 22:15:00.385747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.385718 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vhp\" (UniqueName: \"kubernetes.io/projected/9a325e37-9e3c-4828-8a41-f88ed2a4acda-kube-api-access-g4vhp\") pod \"maas-api-key-cleanup-29889975-4pxcw\" (UID: \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:00.427689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.427654 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b959\" (UniqueName: \"kubernetes.io/projected/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-kube-api-access-4b959\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws\" (UID: \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:00.427856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.427775 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfct\" (UniqueName: \"kubernetes.io/projected/0b16d139-d87a-44b3-9e53-1f2764e561c7-kube-api-access-mpfct\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk\" (UID: \"0b16d139-d87a-44b3-9e53-1f2764e561c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:00.427856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.427814 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b16d139-d87a-44b3-9e53-1f2764e561c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk\" (UID: \"0b16d139-d87a-44b3-9e53-1f2764e561c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:00.427979 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.427903 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws\" (UID: \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:00.428498 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.428478 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b16d139-d87a-44b3-9e53-1f2764e561c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk\" (UID: \"0b16d139-d87a-44b3-9e53-1f2764e561c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:00.428577 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.428514 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws\" (UID: \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:00.487265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.487171 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfct\" (UniqueName: \"kubernetes.io/projected/0b16d139-d87a-44b3-9e53-1f2764e561c7-kube-api-access-mpfct\") pod \"maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk\" (UID: \"0b16d139-d87a-44b3-9e53-1f2764e561c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:00.487265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.487171 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b959\" (UniqueName: \"kubernetes.io/projected/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-kube-api-access-4b959\") pod \"maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws\" (UID: \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:00.503951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.503928 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:00.525136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.525112 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:00.532496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.532472 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:00.876525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.876451 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw"] Oct 30 22:15:00.882366 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:15:00.882340 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a325e37_9e3c_4828_8a41_f88ed2a4acda.slice/crio-621da80f410286379657db6d503b33927b39152cafc5140f3edd42eaacb95a7b WatchSource:0}: Error finding container 621da80f410286379657db6d503b33927b39152cafc5140f3edd42eaacb95a7b: Status 404 returned error can't find the container with id 621da80f410286379657db6d503b33927b39152cafc5140f3edd42eaacb95a7b Oct 30 22:15:00.926660 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.926634 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk"] Oct 30 22:15:00.930375 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:00.930354 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws"] Oct 30 22:15:00.932432 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:15:00.932404 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b16d139_d87a_44b3_9e53_1f2764e561c7.slice/crio-d17502c20316697661f3c8ee901ebff6b92054b97943ae96bc1b206ffad42ba1 WatchSource:0}: Error finding container d17502c20316697661f3c8ee901ebff6b92054b97943ae96bc1b206ffad42ba1: Status 404 returned error can't find the container with id d17502c20316697661f3c8ee901ebff6b92054b97943ae96bc1b206ffad42ba1 Oct 30 22:15:00.935465 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:15:00.935439 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d7c675_12f9_4e0e_8308_fd07a0bb0e52.slice/crio-4f5fa57bfee46883a8937825e3c8971ce5495cebb0edc3b448b9f2df2f447454 WatchSource:0}: Error finding container 4f5fa57bfee46883a8937825e3c8971ce5495cebb0edc3b448b9f2df2f447454: Status 404 returned error can't find the container with id 4f5fa57bfee46883a8937825e3c8971ce5495cebb0edc3b448b9f2df2f447454 Oct 30 22:15:01.637396 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:01.637356 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" event={"ID":"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52","Type":"ContainerStarted","Data":"4f5fa57bfee46883a8937825e3c8971ce5495cebb0edc3b448b9f2df2f447454"} Oct 30 22:15:01.638604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:01.638575 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" event={"ID":"9a325e37-9e3c-4828-8a41-f88ed2a4acda","Type":"ContainerStarted","Data":"621da80f410286379657db6d503b33927b39152cafc5140f3edd42eaacb95a7b"} Oct 30 22:15:01.639783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:01.639761 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" event={"ID":"0b16d139-d87a-44b3-9e53-1f2764e561c7","Type":"ContainerStarted","Data":"d17502c20316697661f3c8ee901ebff6b92054b97943ae96bc1b206ffad42ba1"} Oct 30 22:15:05.374665 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.374638 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv"] Oct 30 22:15:05.379875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.379852 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.454556 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.454531 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"premium-w3-23f3d4-kserve-self-signed-certs\"" Oct 30 22:15:05.456800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.456779 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv"] Oct 30 22:15:05.488905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.488878 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-home\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.489051 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.488921 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18303bc1-aa75-4a32-9173-a546eced5ba9-tls-certs\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.489051 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.488941 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwh6x\" (UniqueName: \"kubernetes.io/projected/18303bc1-aa75-4a32-9173-a546eced5ba9-kube-api-access-gwh6x\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.489148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.489052 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-tmp-dir\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.489148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.489095 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-model-cache\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.489148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.489127 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-dshm\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.590853 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.590820 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-home\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.590970 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.590878 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18303bc1-aa75-4a32-9173-a546eced5ba9-tls-certs\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.591026 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.590997 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwh6x\" (UniqueName: \"kubernetes.io/projected/18303bc1-aa75-4a32-9173-a546eced5ba9-kube-api-access-gwh6x\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.591063 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.591046 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-tmp-dir\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.591108 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.591099 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-model-cache\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.591156 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.591129 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-dshm\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.591230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.591193 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-home\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.591449 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.591418 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-tmp-dir\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.591580 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.591552 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-model-cache\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.593325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.593299 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-dshm\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.593425 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.593393 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18303bc1-aa75-4a32-9173-a546eced5ba9-tls-certs\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.650434 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.650377 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwh6x\" (UniqueName: \"kubernetes.io/projected/18303bc1-aa75-4a32-9173-a546eced5ba9-kube-api-access-gwh6x\") pod \"premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:05.763372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.763342 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:15:05.846429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:05.846405 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:06.216563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.216482 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv"] Oct 30 22:15:06.221360 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:15:06.221335 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18303bc1_aa75_4a32_9173_a546eced5ba9.slice/crio-65abeef86c7ca908d0c46e596ae11266d86436a1f6c62d37ab5817c2f767ba2f WatchSource:0}: Error finding container 65abeef86c7ca908d0c46e596ae11266d86436a1f6c62d37ab5817c2f767ba2f: Status 404 returned error can't find the container with id 65abeef86c7ca908d0c46e596ae11266d86436a1f6c62d37ab5817c2f767ba2f Oct 30 22:15:06.676076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.676040 5226 generic.go:356] "Generic (PLEG): container finished" podID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerID="b96f58f5d283d83e94ee4fd8f90ceb062bce522bedda3a1c790bcda064320510" exitCode=60 Oct 30 22:15:06.676537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.676108 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" event={"ID":"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52","Type":"ContainerDied","Data":"b96f58f5d283d83e94ee4fd8f90ceb062bce522bedda3a1c790bcda064320510"} Oct 30 22:15:06.676537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.676360 5226 scope.go:122] "RemoveContainer" containerID="b96f58f5d283d83e94ee4fd8f90ceb062bce522bedda3a1c790bcda064320510" Oct 30 22:15:06.677868 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.677846 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" event={"ID":"18303bc1-aa75-4a32-9173-a546eced5ba9","Type":"ContainerStarted","Data":"808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c"} Oct 30 22:15:06.677995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.677879 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" event={"ID":"18303bc1-aa75-4a32-9173-a546eced5ba9","Type":"ContainerStarted","Data":"65abeef86c7ca908d0c46e596ae11266d86436a1f6c62d37ab5817c2f767ba2f"} Oct 30 22:15:06.679448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.679427 5226 generic.go:356] "Generic (PLEG): container finished" podID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerID="957ceaf748a006e88b5cd7d422e8eb6137a512f86e26c9f950b7ac3ce5b2f563" exitCode=60 Oct 30 22:15:06.679547 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.679484 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" event={"ID":"9a325e37-9e3c-4828-8a41-f88ed2a4acda","Type":"ContainerDied","Data":"957ceaf748a006e88b5cd7d422e8eb6137a512f86e26c9f950b7ac3ce5b2f563"} Oct 30 22:15:06.679669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.679652 5226 scope.go:122] "RemoveContainer" containerID="957ceaf748a006e88b5cd7d422e8eb6137a512f86e26c9f950b7ac3ce5b2f563" Oct 30 22:15:06.681186 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.681163 5226 generic.go:356] "Generic (PLEG): container finished" podID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerID="9a5eb31ed31083e49f2bd209841c01c9574a571e61a2451c3f704fa6fa17ca26" exitCode=60 Oct 30 22:15:06.681311 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.681252 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" event={"ID":"0b16d139-d87a-44b3-9e53-1f2764e561c7","Type":"ContainerDied","Data":"9a5eb31ed31083e49f2bd209841c01c9574a571e61a2451c3f704fa6fa17ca26"} Oct 30 22:15:06.681450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.681433 5226 scope.go:122] "RemoveContainer" containerID="9a5eb31ed31083e49f2bd209841c01c9574a571e61a2451c3f704fa6fa17ca26" Oct 30 22:15:06.847143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:06.847057 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:07.135671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.135613 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podStartSLOduration=2.135594483 podStartE2EDuration="2.135594483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.132952135 +0000 UTC m=+798.390293870" watchObservedRunningTime="2026-10-30 22:15:07.135594483 +0000 UTC m=+798.392936194" Oct 30 22:15:07.688921 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.688885 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" event={"ID":"9a325e37-9e3c-4828-8a41-f88ed2a4acda","Type":"ContainerStarted","Data":"976aeae09818de41db10b031fde2fa87ef38ed6c44eb2f58dccf830955d886f5"} Oct 30 22:15:07.690774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.690749 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" event={"ID":"0b16d139-d87a-44b3-9e53-1f2764e561c7","Type":"ContainerStarted","Data":"6557ed93f7fda354e497ec47c490d5d539732745fdb0e89d33c212e48bc8db39"} Oct 30 22:15:07.692624 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.692601 5226 generic.go:356] "Generic (PLEG): container finished" podID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerID="875061a3d34279a5db2c414b20db673fbc776ad2b81f3fb0cf92384ed14e381a" exitCode=60 Oct 30 22:15:07.692717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.692677 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" event={"ID":"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52","Type":"ContainerDied","Data":"875061a3d34279a5db2c414b20db673fbc776ad2b81f3fb0cf92384ed14e381a"} Oct 30 22:15:07.692770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.692727 5226 scope.go:122] "RemoveContainer" containerID="b96f58f5d283d83e94ee4fd8f90ceb062bce522bedda3a1c790bcda064320510" Oct 30 22:15:07.692853 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.692834 5226 scope.go:122] "RemoveContainer" containerID="875061a3d34279a5db2c414b20db673fbc776ad2b81f3fb0cf92384ed14e381a" Oct 30 22:15:07.693090 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:07.693068 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws_odh-ai-gateway-infra(b4d7c675-12f9-4e0e-8308-fd07a0bb0e52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" Oct 30 22:15:07.800704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.800654 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" podStartSLOduration=3.328533661 podStartE2EDuration="7.80064001s" totalImagesPullingTime="4.472106349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.798770198 +0000 UTC m=+799.056111920" watchObservedRunningTime="2026-10-30 22:15:07.80064001 +0000 UTC m=+799.057981721" Oct 30 22:15:07.907388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:07.907272 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" podStartSLOduration=3.490133589 podStartE2EDuration="7.907256995s" totalImagesPullingTime="4.417123406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:07.907193244 +0000 UTC m=+799.164534965" watchObservedRunningTime="2026-10-30 22:15:07.907256995 +0000 UTC m=+799.164598706" Oct 30 22:15:08.699628 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.699596 5226 scope.go:122] "RemoveContainer" containerID="875061a3d34279a5db2c414b20db673fbc776ad2b81f3fb0cf92384ed14e381a" Oct 30 22:15:08.700088 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:08.699875 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws_odh-ai-gateway-infra(b4d7c675-12f9-4e0e-8308-fd07a0bb0e52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" Oct 30 22:15:08.701328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.701302 5226 generic.go:356] "Generic (PLEG): container finished" podID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerID="976aeae09818de41db10b031fde2fa87ef38ed6c44eb2f58dccf830955d886f5" exitCode=60 Oct 30 22:15:08.701460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.701393 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" event={"ID":"9a325e37-9e3c-4828-8a41-f88ed2a4acda","Type":"ContainerDied","Data":"976aeae09818de41db10b031fde2fa87ef38ed6c44eb2f58dccf830955d886f5"} Oct 30 22:15:08.701537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.701497 5226 scope.go:122] "RemoveContainer" containerID="957ceaf748a006e88b5cd7d422e8eb6137a512f86e26c9f950b7ac3ce5b2f563" Oct 30 22:15:08.701590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.701554 5226 scope.go:122] "RemoveContainer" containerID="976aeae09818de41db10b031fde2fa87ef38ed6c44eb2f58dccf830955d886f5" Oct 30 22:15:08.701816 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:08.701791 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-4pxcw_odh-ai-gateway-infra(9a325e37-9e3c-4828-8a41-f88ed2a4acda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" Oct 30 22:15:08.703187 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.703166 5226 generic.go:356] "Generic (PLEG): container finished" podID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerID="6557ed93f7fda354e497ec47c490d5d539732745fdb0e89d33c212e48bc8db39" exitCode=60 Oct 30 22:15:08.703271 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.703235 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" event={"ID":"0b16d139-d87a-44b3-9e53-1f2764e561c7","Type":"ContainerDied","Data":"6557ed93f7fda354e497ec47c490d5d539732745fdb0e89d33c212e48bc8db39"} Oct 30 22:15:08.703494 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.703478 5226 scope.go:122] "RemoveContainer" containerID="6557ed93f7fda354e497ec47c490d5d539732745fdb0e89d33c212e48bc8db39" Oct 30 22:15:08.703744 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:08.703728 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk_odh-ai-gateway-infra(0b16d139-d87a-44b3-9e53-1f2764e561c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" Oct 30 22:15:08.712974 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:08.712957 5226 scope.go:122] "RemoveContainer" containerID="9a5eb31ed31083e49f2bd209841c01c9574a571e61a2451c3f704fa6fa17ca26" Oct 30 22:15:09.711774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:09.711745 5226 scope.go:122] "RemoveContainer" containerID="976aeae09818de41db10b031fde2fa87ef38ed6c44eb2f58dccf830955d886f5" Oct 30 22:15:09.712272 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:09.712049 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-4pxcw_odh-ai-gateway-infra(9a325e37-9e3c-4828-8a41-f88ed2a4acda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" Oct 30 22:15:09.713306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:09.713285 5226 scope.go:122] "RemoveContainer" containerID="6557ed93f7fda354e497ec47c490d5d539732745fdb0e89d33c212e48bc8db39" Oct 30 22:15:09.713538 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:09.713518 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk_odh-ai-gateway-infra(0b16d139-d87a-44b3-9e53-1f2764e561c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" Oct 30 22:15:15.847195 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:15.847157 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:15.860344 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:15.860317 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:16.760622 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:16.760591 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:15:18.903955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:18.901603 5226 scope.go:122] "RemoveContainer" containerID="6557ed93f7fda354e497ec47c490d5d539732745fdb0e89d33c212e48bc8db39" Oct 30 22:15:19.768712 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:19.768678 5226 generic.go:356] "Generic (PLEG): container finished" podID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerID="a86bc524196a593f30266c4a877c4b2d0c9ec7e11396f6e4ef6ffc08e138f770" exitCode=60 Oct 30 22:15:19.768712 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:19.768704 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" event={"ID":"0b16d139-d87a-44b3-9e53-1f2764e561c7","Type":"ContainerDied","Data":"a86bc524196a593f30266c4a877c4b2d0c9ec7e11396f6e4ef6ffc08e138f770"} Oct 30 22:15:19.768925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:19.768794 5226 scope.go:122] "RemoveContainer" containerID="6557ed93f7fda354e497ec47c490d5d539732745fdb0e89d33c212e48bc8db39" Oct 30 22:15:19.769150 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:19.769130 5226 scope.go:122] "RemoveContainer" containerID="a86bc524196a593f30266c4a877c4b2d0c9ec7e11396f6e4ef6ffc08e138f770" Oct 30 22:15:19.769470 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:19.769447 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk_odh-ai-gateway-infra(0b16d139-d87a-44b3-9e53-1f2764e561c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" Oct 30 22:15:20.890097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:20.890054 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk"] Oct 30 22:15:20.898310 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:20.898274 5226 scope.go:122] "RemoveContainer" containerID="875061a3d34279a5db2c414b20db673fbc776ad2b81f3fb0cf92384ed14e381a" Oct 30 22:15:21.031187 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.031162 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:21.152304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.152262 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b16d139-d87a-44b3-9e53-1f2764e561c7-kube-api-access-mpfct\" (UniqueName: \"kubernetes.io/projected/0b16d139-d87a-44b3-9e53-1f2764e561c7-kube-api-access-mpfct\") pod \"0b16d139-d87a-44b3-9e53-1f2764e561c7\" (UID: \"0b16d139-d87a-44b3-9e53-1f2764e561c7\") " Oct 30 22:15:21.152476 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.152381 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b16d139-d87a-44b3-9e53-1f2764e561c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b16d139-d87a-44b3-9e53-1f2764e561c7-maas-api-service-ca\") pod \"0b16d139-d87a-44b3-9e53-1f2764e561c7\" (UID: \"0b16d139-d87a-44b3-9e53-1f2764e561c7\") " Oct 30 22:15:21.152986 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.152955 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b16d139-d87a-44b3-9e53-1f2764e561c7-maas-api-service-ca" pod "0b16d139-d87a-44b3-9e53-1f2764e561c7" (UID: "0b16d139-d87a-44b3-9e53-1f2764e561c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.154538 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.154506 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b16d139-d87a-44b3-9e53-1f2764e561c7-kube-api-access-mpfct" pod "0b16d139-d87a-44b3-9e53-1f2764e561c7" (UID: "0b16d139-d87a-44b3-9e53-1f2764e561c7"). InnerVolumeSpecName "kube-api-access-mpfct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.254197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.254166 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b16d139-d87a-44b3-9e53-1f2764e561c7-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.254197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.254197 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfct\" (UniqueName: \"kubernetes.io/projected/0b16d139-d87a-44b3-9e53-1f2764e561c7-kube-api-access-mpfct\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.782587 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.782554 5226 generic.go:356] "Generic (PLEG): container finished" podID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerID="1de6968cb9fdda24892e56e14e37a5f8e8e2c935d6023e30ae63779b04f92806" exitCode=60 Oct 30 22:15:21.782750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.782623 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" event={"ID":"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52","Type":"ContainerDied","Data":"1de6968cb9fdda24892e56e14e37a5f8e8e2c935d6023e30ae63779b04f92806"} Oct 30 22:15:21.782750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.782680 5226 scope.go:122] "RemoveContainer" containerID="875061a3d34279a5db2c414b20db673fbc776ad2b81f3fb0cf92384ed14e381a" Oct 30 22:15:21.783037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.783016 5226 scope.go:122] "RemoveContainer" containerID="1de6968cb9fdda24892e56e14e37a5f8e8e2c935d6023e30ae63779b04f92806" Oct 30 22:15:21.783332 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:21.783313 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws_odh-ai-gateway-infra(b4d7c675-12f9-4e0e-8308-fd07a0bb0e52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" Oct 30 22:15:21.784053 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.784037 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" Oct 30 22:15:21.784143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.784067 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk" event={"ID":"0b16d139-d87a-44b3-9e53-1f2764e561c7","Type":"ContainerDied","Data":"d17502c20316697661f3c8ee901ebff6b92054b97943ae96bc1b206ffad42ba1"} Oct 30 22:15:21.794442 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.794424 5226 scope.go:122] "RemoveContainer" containerID="a86bc524196a593f30266c4a877c4b2d0c9ec7e11396f6e4ef6ffc08e138f770" Oct 30 22:15:21.898381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:21.898347 5226 scope.go:122] "RemoveContainer" containerID="976aeae09818de41db10b031fde2fa87ef38ed6c44eb2f58dccf830955d886f5" Oct 30 22:15:22.010569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:22.010488 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk"] Oct 30 22:15:22.017749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:22.017718 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f80a0e-29889975-2bdzk"] Oct 30 22:15:22.792441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:22.792414 5226 generic.go:356] "Generic (PLEG): container finished" podID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerID="944985f9d2510daf8dfb9429afb96ad42e3e3a2e8020814f977be9cd1064df63" exitCode=60 Oct 30 22:15:22.792566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:22.792488 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" event={"ID":"9a325e37-9e3c-4828-8a41-f88ed2a4acda","Type":"ContainerDied","Data":"944985f9d2510daf8dfb9429afb96ad42e3e3a2e8020814f977be9cd1064df63"} Oct 30 22:15:22.792566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:22.792558 5226 scope.go:122] "RemoveContainer" containerID="976aeae09818de41db10b031fde2fa87ef38ed6c44eb2f58dccf830955d886f5" Oct 30 22:15:22.792940 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:22.792917 5226 scope.go:122] "RemoveContainer" containerID="944985f9d2510daf8dfb9429afb96ad42e3e3a2e8020814f977be9cd1064df63" Oct 30 22:15:22.793237 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:22.793200 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-4pxcw_odh-ai-gateway-infra(9a325e37-9e3c-4828-8a41-f88ed2a4acda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" Oct 30 22:15:22.904868 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:22.904776 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" path="/var/lib/kubelet/pods/0b16d139-d87a-44b3-9e53-1f2764e561c7/volumes" Oct 30 22:15:22.918310 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:22.918282 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws"] Oct 30 22:15:23.065313 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.065285 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:23.173462 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.173435 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-kube-api-access-4b959\" (UniqueName: \"kubernetes.io/projected/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-kube-api-access-4b959\") pod \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\" (UID: \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\") " Oct 30 22:15:23.173619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.173478 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-maas-api-service-ca\") pod \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\" (UID: \"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52\") " Oct 30 22:15:23.173972 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.173941 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-maas-api-service-ca" pod "b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" (UID: "b4d7c675-12f9-4e0e-8308-fd07a0bb0e52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:23.174132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.174115 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.175829 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.175793 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-kube-api-access-4b959" pod "b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" (UID: "b4d7c675-12f9-4e0e-8308-fd07a0bb0e52"). InnerVolumeSpecName "kube-api-access-4b959". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:23.274762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.274716 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b959\" (UniqueName: \"kubernetes.io/projected/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52-kube-api-access-4b959\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.800286 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.800259 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" Oct 30 22:15:23.800286 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.800271 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws" event={"ID":"b4d7c675-12f9-4e0e-8308-fd07a0bb0e52","Type":"ContainerDied","Data":"4f5fa57bfee46883a8937825e3c8971ce5495cebb0edc3b448b9f2df2f447454"} Oct 30 22:15:23.800493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.800362 5226 scope.go:122] "RemoveContainer" containerID="1de6968cb9fdda24892e56e14e37a5f8e8e2c935d6023e30ae63779b04f92806" Oct 30 22:15:23.921348 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.921314 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw"] Oct 30 22:15:23.963987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.963955 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws"] Oct 30 22:15:23.968715 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:23.968683 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b13755-29889975-rh8ws"] Oct 30 22:15:24.058472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058409 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w"] Oct 30 22:15:24.058846 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058833 5226 cpu_manager.go:414] "containerMap: removing container" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:24.058883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058847 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:24.058883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058861 5226 cpu_manager.go:414] "containerMap: removing container" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.058883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058865 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.058883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058872 5226 cpu_manager.go:414] "containerMap: removing container" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:24.058883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058876 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:24.058883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058885 5226 cpu_manager.go:414] "containerMap: removing container" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.059056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058889 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.059056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058982 5226 cpu_manager.go:414] "containerMap: removing container" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.059056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.058988 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.059148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.059076 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.059148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.059084 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:24.059148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.059091 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.059148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.059099 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" containerName="cleanup" Oct 30 22:15:24.059148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.059106 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:24.063920 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.063900 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.064699 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.064676 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:24.113604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.113570 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"premium-w5-f80a0e-kserve-self-signed-certs\"" Oct 30 22:15:24.122955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.122926 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w"] Oct 30 22:15:24.183452 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.183415 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a325e37-9e3c-4828-8a41-f88ed2a4acda-kube-api-access-g4vhp\" (UniqueName: \"kubernetes.io/projected/9a325e37-9e3c-4828-8a41-f88ed2a4acda-kube-api-access-g4vhp\") pod \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\" (UID: \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\") " Oct 30 22:15:24.183641 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.183478 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a325e37-9e3c-4828-8a41-f88ed2a4acda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a325e37-9e3c-4828-8a41-f88ed2a4acda-maas-api-service-ca\") pod \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\" (UID: \"9a325e37-9e3c-4828-8a41-f88ed2a4acda\") " Oct 30 22:15:24.184243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.183795 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-home\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.184243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.183889 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-model-cache\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.184243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.183920 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a325e37-9e3c-4828-8a41-f88ed2a4acda-maas-api-service-ca" pod "9a325e37-9e3c-4828-8a41-f88ed2a4acda" (UID: "9a325e37-9e3c-4828-8a41-f88ed2a4acda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:24.184243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.184027 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmt7\" (UniqueName: \"kubernetes.io/projected/9253bf18-33e8-4e12-8b40-6a39bc59abbe-kube-api-access-zxmt7\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.184243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.184067 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-dshm\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.184243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.184170 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tls-certs\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.184517 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.184261 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tmp-dir\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.184517 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.184367 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a325e37-9e3c-4828-8a41-f88ed2a4acda-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.185591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.185569 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a325e37-9e3c-4828-8a41-f88ed2a4acda-kube-api-access-g4vhp" pod "9a325e37-9e3c-4828-8a41-f88ed2a4acda" (UID: "9a325e37-9e3c-4828-8a41-f88ed2a4acda"). InnerVolumeSpecName "kube-api-access-g4vhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:24.285750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.285717 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-dshm\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.285937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.285772 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tls-certs\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.285937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.285816 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tmp-dir\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.285937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.285851 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-home\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.285937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.285884 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-model-cache\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.286159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.285960 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmt7\" (UniqueName: \"kubernetes.io/projected/9253bf18-33e8-4e12-8b40-6a39bc59abbe-kube-api-access-zxmt7\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.286159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.286032 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vhp\" (UniqueName: \"kubernetes.io/projected/9a325e37-9e3c-4828-8a41-f88ed2a4acda-kube-api-access-g4vhp\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.286281 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.286258 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tmp-dir\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.286346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.286319 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-model-cache\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.286399 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.286347 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-home\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.288068 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.288040 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-dshm\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.288379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.288360 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tls-certs\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.344196 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.344121 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmt7\" (UniqueName: \"kubernetes.io/projected/9253bf18-33e8-4e12-8b40-6a39bc59abbe-kube-api-access-zxmt7\") pod \"premium-w5-f80a0e-kserve-7887f6bc67-6527w\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.375035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.374999 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:24.421881 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:24.420614 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:15:24.752400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.752366 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w"] Oct 30 22:15:24.758624 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:15:24.758591 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9253bf18_33e8_4e12_8b40_6a39bc59abbe.slice/crio-9ace0002c11c90e1bbb09cd6262c9cc8387e9884dcf41ba25fb04d4292ad5f28 WatchSource:0}: Error finding container 9ace0002c11c90e1bbb09cd6262c9cc8387e9884dcf41ba25fb04d4292ad5f28: Status 404 returned error can't find the container with id 9ace0002c11c90e1bbb09cd6262c9cc8387e9884dcf41ba25fb04d4292ad5f28 Oct 30 22:15:24.808869 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.808846 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" Oct 30 22:15:24.809040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.808845 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw" event={"ID":"9a325e37-9e3c-4828-8a41-f88ed2a4acda","Type":"ContainerDied","Data":"621da80f410286379657db6d503b33927b39152cafc5140f3edd42eaacb95a7b"} Oct 30 22:15:24.809040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.808978 5226 scope.go:122] "RemoveContainer" containerID="944985f9d2510daf8dfb9429afb96ad42e3e3a2e8020814f977be9cd1064df63" Oct 30 22:15:24.810791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.810757 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" event={"ID":"9253bf18-33e8-4e12-8b40-6a39bc59abbe","Type":"ContainerStarted","Data":"9ace0002c11c90e1bbb09cd6262c9cc8387e9884dcf41ba25fb04d4292ad5f28"} Oct 30 22:15:24.904960 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.904929 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d7c675-12f9-4e0e-8308-fd07a0bb0e52" path="/var/lib/kubelet/pods/b4d7c675-12f9-4e0e-8308-fd07a0bb0e52/volumes" Oct 30 22:15:24.969595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.969564 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw"] Oct 30 22:15:24.972074 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:24.972047 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-4pxcw"] Oct 30 22:15:25.820085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:25.820056 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" event={"ID":"9253bf18-33e8-4e12-8b40-6a39bc59abbe","Type":"ContainerStarted","Data":"2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe"} Oct 30 22:15:25.932472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:25.932410 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podStartSLOduration=1.9323963050000001 podStartE2EDuration="1.932396305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:25.930424837 +0000 UTC m=+817.187766547" watchObservedRunningTime="2026-10-30 22:15:25.932396305 +0000 UTC m=+817.189738014" Oct 30 22:15:26.375763 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:26.375723 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:26.910647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:26.910611 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" path="/var/lib/kubelet/pods/9a325e37-9e3c-4828-8a41-f88ed2a4acda/volumes" Oct 30 22:15:26.991839 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:26.991801 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:15:31.015521 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.015439 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r"] Oct 30 22:15:31.015912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.015875 5226 cpu_manager.go:414] "containerMap: removing container" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:15:31.015912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.015885 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:15:31.015912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.015914 5226 cpu_manager.go:414] "containerMap: removing container" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:31.016031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.015920 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:31.016031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.015930 5226 cpu_manager.go:414] "containerMap: removing container" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:15:31.016031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.015936 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:15:31.016031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.016006 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b16d139-d87a-44b3-9e53-1f2764e561c7" containerName="cleanup" Oct 30 22:15:31.016031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.016015 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:15:31.016031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.016024 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:15:31.020875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.020854 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057528 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113dec97-8335-482f-8a29-bd6bb0c04894-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057711 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057585 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057711 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057648 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftqwf\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-kube-api-access-ftqwf\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057711 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057685 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057713 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057804 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057842 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113dec97-8335-482f-8a29-bd6bb0c04894-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057881 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.057955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.057920 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.070015 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.069991 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-af30a9-openshift-default-dockercfg-fzzwk\"" Oct 30 22:15:31.118795 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.118765 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r"] Oct 30 22:15:31.158443 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158412 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158460 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158485 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158510 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113dec97-8335-482f-8a29-bd6bb0c04894-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158541 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158558 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158605 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113dec97-8335-482f-8a29-bd6bb0c04894-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158802 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158810 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158852 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.158987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.158875 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftqwf\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-kube-api-access-ftqwf\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.159246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.159086 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.159246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.159148 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.159399 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.159265 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113dec97-8335-482f-8a29-bd6bb0c04894-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.160828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.160809 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.161138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.161117 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113dec97-8335-482f-8a29-bd6bb0c04894-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.216537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.216505 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.216695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.216559 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftqwf\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-kube-api-access-ftqwf\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.335847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.335759 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:31.765423 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.765386 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r"] Oct 30 22:15:31.771756 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:15:31.771728 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113dec97_8335_482f_8a29_bd6bb0c04894.slice/crio-2065a94b80acc494709f20b84fa44c57d4bedcddb0582709da5bcc9dd05cb515 WatchSource:0}: Error finding container 2065a94b80acc494709f20b84fa44c57d4bedcddb0582709da5bcc9dd05cb515: Status 404 returned error can't find the container with id 2065a94b80acc494709f20b84fa44c57d4bedcddb0582709da5bcc9dd05cb515 Oct 30 22:15:31.773893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.773863 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:15:31.773969 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.773928 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:15:31.773969 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.773958 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:15:31.855535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.855489 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" event={"ID":"113dec97-8335-482f-8a29-bd6bb0c04894","Type":"ContainerStarted","Data":"f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d"} Oct 30 22:15:31.855716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.855556 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" event={"ID":"113dec97-8335-482f-8a29-bd6bb0c04894","Type":"ContainerStarted","Data":"2065a94b80acc494709f20b84fa44c57d4bedcddb0582709da5bcc9dd05cb515"} Oct 30 22:15:31.969133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:31.969082 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podStartSLOduration=1.969068561 podStartE2EDuration="1.969068561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:31.968859686 +0000 UTC m=+823.226201397" watchObservedRunningTime="2026-10-30 22:15:31.969068561 +0000 UTC m=+823.226410270" Oct 30 22:15:32.344687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:32.344601 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:32.348943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:32.348905 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:15:32.349152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:32.349033 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 22:15:33.347229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:33.341742 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:15:33.347229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:33.341804 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 22:15:34.342675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:34.342633 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:15:34.343705 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:34.343670 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 22:15:34.375426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:34.375342 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:34.388555 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:34.388523 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:34.883870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:34.883843 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:15:35.345635 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:35.345594 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:15:35.345932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:35.345898 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 22:15:36.348229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:36.345311 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:15:36.348229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:36.345383 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 22:15:37.348691 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:37.348655 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 22:15:37.349234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:37.348722 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 22:15:39.086543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:39.086501 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:39.087286 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:39.087256 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:39.088720 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:39.088689 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:15:54.410614 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:15:54.410571 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:15:54.659258 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:15:54.659216 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:16:24.411868 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:16:24.411831 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:16:25.690997 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:25.690960 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:16:26.046105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:26.046023 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz"] Oct 30 22:16:26.046301 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:26.046263 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" containerID="cri-o://b386f522932713a12b2ea2cf15c562fb1366e02bb91bc692278261df3442c8e2" gracePeriod=30 Oct 30 22:16:31.188964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.188925 5226 generic.go:356] "Generic (PLEG): container finished" podID="3a989905-c9be-4423-8d8c-83400f966607" containerID="b386f522932713a12b2ea2cf15c562fb1366e02bb91bc692278261df3442c8e2" exitCode=0 Oct 30 22:16:31.189459 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.188992 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" event={"ID":"3a989905-c9be-4423-8d8c-83400f966607","Type":"ContainerDied","Data":"b386f522932713a12b2ea2cf15c562fb1366e02bb91bc692278261df3442c8e2"} Oct 30 22:16:31.189459 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.189069 5226 scope.go:122] "RemoveContainer" containerID="55558ea82e0caff68e3ca06b526f67413f279e8fd002004e31d77e2ec2c62aff" Oct 30 22:16:31.305682 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.305658 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:16:31.405283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405193 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-envoy\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405314 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-certs\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405358 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a989905-c9be-4423-8d8c-83400f966607-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a989905-c9be-4423-8d8c-83400f966607-istiod-ca-cert\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405402 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-istio-token\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-istio-token\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405519 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-data\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405568 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-credential-socket\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405598 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-kube-api-access-4ztjt\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-kube-api-access-4ztjt\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405630 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-socket\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405639 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-certs" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:31.405783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405662 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a989905-c9be-4423-8d8c-83400f966607-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a989905-c9be-4423-8d8c-83400f966607-istio-podinfo\") pod \"3a989905-c9be-4423-8d8c-83400f966607\" (UID: \"3a989905-c9be-4423-8d8c-83400f966607\") " Oct 30 22:16:31.405950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405798 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a989905-c9be-4423-8d8c-83400f966607-istiod-ca-cert" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:31.405950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.405807 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-credential-socket" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:31.406058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.406010 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-data" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:31.406112 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.406071 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.406112 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.406094 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a989905-c9be-4423-8d8c-83400f966607-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.406249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.406114 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.406249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.406092 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-socket" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:31.407586 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.407562 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-envoy" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:31.407692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.407653 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-kube-api-access-4ztjt" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "kube-api-access-4ztjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:31.407735 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.407725 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-istio-token" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:31.407817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.407798 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a989905-c9be-4423-8d8c-83400f966607-istio-podinfo" pod "3a989905-c9be-4423-8d8c-83400f966607" (UID: "3a989905-c9be-4423-8d8c-83400f966607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:31.506814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.506781 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.506955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.506817 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ztjt\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-kube-api-access-4ztjt\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.506955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.506834 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.506955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.506848 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a989905-c9be-4423-8d8c-83400f966607-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.506955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.506862 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a989905-c9be-4423-8d8c-83400f966607-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:31.506955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:31.506876 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a989905-c9be-4423-8d8c-83400f966607-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.196008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:32.195975 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" Oct 30 22:16:32.196427 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:32.196008 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz" event={"ID":"3a989905-c9be-4423-8d8c-83400f966607","Type":"ContainerDied","Data":"da46e8248b02f9bc9f9a2599b263208313cdab48f2faec7f2211b44a46822930"} Oct 30 22:16:32.196427 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:32.196067 5226 scope.go:122] "RemoveContainer" containerID="b386f522932713a12b2ea2cf15c562fb1366e02bb91bc692278261df3442c8e2" Oct 30 22:16:32.418925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:32.418896 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz"] Oct 30 22:16:32.465570 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:32.465493 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nmwcz"] Oct 30 22:16:32.904131 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:32.904052 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a989905-c9be-4423-8d8c-83400f966607" path="/var/lib/kubelet/pods/3a989905-c9be-4423-8d8c-83400f966607/volumes" Oct 30 22:16:47.685018 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.684979 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7"] Oct 30 22:16:47.685535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685419 5226 cpu_manager.go:414] "containerMap: removing container" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" Oct 30 22:16:47.685535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685431 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" Oct 30 22:16:47.685535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685452 5226 cpu_manager.go:414] "containerMap: removing container" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:16:47.685535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685457 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:16:47.685535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685470 5226 cpu_manager.go:414] "containerMap: removing container" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" Oct 30 22:16:47.685535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685475 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" Oct 30 22:16:47.685756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685553 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" Oct 30 22:16:47.685756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685565 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a989905-c9be-4423-8d8c-83400f966607" containerName="istio-proxy" Oct 30 22:16:47.685756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.685571 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a325e37-9e3c-4828-8a41-f88ed2a4acda" containerName="cleanup" Oct 30 22:16:47.689231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.689192 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.738690 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.738664 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f80a0e-openshift-default-dockercfg-h5r64\"" Oct 30 22:16:47.762506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762475 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.762506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762509 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.762696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762534 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.762696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762648 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.762773 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762697 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.762773 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762750 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.762847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762789 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.762847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762834 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-kube-api-access-hbn2h\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.762936 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.762919 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.795434 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.795403 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7"] Oct 30 22:16:47.864447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864413 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-kube-api-access-hbn2h\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.864653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864469 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.864653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864520 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.864653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864538 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.864653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864553 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.864653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864583 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.864653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864637 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.864956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864665 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.864956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864706 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.865060 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.864957 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.865137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.865118 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.865199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.865151 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.865291 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.865234 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.865485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.865464 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.866875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.866852 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.867007 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.866987 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.922753 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.922711 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:47.922922 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:47.922841 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-kube-api-access-hbn2h\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:48.001811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:48.001721 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:48.424571 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:48.424534 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7"] Oct 30 22:16:48.431474 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:16:48.431447 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3ed2bd_990d_4800_bdbd_90ac8f2d3ed4.slice/crio-eef1cfcd304c8dbeb13cf7170c852dfd7f1dae7218878ca763285ef5556cccf2 WatchSource:0}: Error finding container eef1cfcd304c8dbeb13cf7170c852dfd7f1dae7218878ca763285ef5556cccf2: Status 404 returned error can't find the container with id eef1cfcd304c8dbeb13cf7170c852dfd7f1dae7218878ca763285ef5556cccf2 Oct 30 22:16:48.434078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:48.434045 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:16:48.434176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:48.434110 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:16:48.434176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:48.434139 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:16:49.010351 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:49.010326 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:16:49.011502 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:49.011479 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:16:49.287252 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:49.287139 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" event={"ID":"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4","Type":"ContainerStarted","Data":"699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59"} Oct 30 22:16:49.287252 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:49.287228 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" event={"ID":"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4","Type":"ContainerStarted","Data":"eef1cfcd304c8dbeb13cf7170c852dfd7f1dae7218878ca763285ef5556cccf2"} Oct 30 22:16:49.458175 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:49.458119 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" podStartSLOduration=2.458106975 podStartE2EDuration="2.458106975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:49.457439515 +0000 UTC m=+900.714781237" watchObservedRunningTime="2026-10-30 22:16:49.458106975 +0000 UTC m=+900.715448751" Oct 30 22:16:50.001902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:50.001867 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:50.003750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:50.003720 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:16:50.003916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:50.003776 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 22:16:51.002719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:51.002679 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:16:51.003139 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:51.002732 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 22:16:52.002137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:52.002100 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:16:52.002350 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:52.002167 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 22:16:53.002819 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:53.002784 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:16:53.003315 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:53.002847 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 22:16:54.002994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:54.002959 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 22:16:54.003408 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:54.003018 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 22:16:54.411042 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:16:54.410957 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:16:55.055742 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:55.055706 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:55.056665 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:55.056636 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:55.057466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:55.057444 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:16:58.113721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:16:58.113683 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:17:24.410939 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:17:24.410864 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:17:26.515532 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.515500 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj"] Oct 30 22:17:26.519764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.519739 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580006 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.579961 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580011 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qws6l\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-kube-api-access-qws6l\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580039 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580159 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580257 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580367 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580352 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fdc873-78c7-404f-b22a-1b78df80c185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580390 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580458 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580564 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580540 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.580727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.580712 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8ctw4\"" Oct 30 22:17:26.628987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.628940 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj"] Oct 30 22:17:26.681864 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.681829 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682026 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.681880 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682026 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.681915 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682026 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.681994 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682038 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qws6l\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-kube-api-access-qws6l\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682068 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682115 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682156 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682263 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fdc873-78c7-404f-b22a-1b78df80c185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682287 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682410 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682476 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682613 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.682888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.682868 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fdc873-78c7-404f-b22a-1b78df80c185-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.684572 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.684536 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.684706 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.684656 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.740377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.740343 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.740517 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.740474 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qws6l\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-kube-api-access-qws6l\") pod \"maas-default-gateway-openshift-default-75bf867b6c-g7jrj\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:26.841113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:26.841023 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:27.268820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:27.268785 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj"] Oct 30 22:17:27.274817 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:17:27.274782 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fdc873_78c7_404f_b22a_1b78df80c185.slice/crio-955dd8b6f3f0abdb316acdeb0fb39c603ba823cd090a9d71b4bd3f5b36ae98b6 WatchSource:0}: Error finding container 955dd8b6f3f0abdb316acdeb0fb39c603ba823cd090a9d71b4bd3f5b36ae98b6: Status 404 returned error can't find the container with id 955dd8b6f3f0abdb316acdeb0fb39c603ba823cd090a9d71b4bd3f5b36ae98b6 Oct 30 22:17:27.277107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:27.277073 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:27.277182 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:27.277142 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:27.277182 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:27.277172 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:27.497219 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:27.497175 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" event={"ID":"c1fdc873-78c7-404f-b22a-1b78df80c185","Type":"ContainerStarted","Data":"5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9"} Oct 30 22:17:27.497399 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:27.497255 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" event={"ID":"c1fdc873-78c7-404f-b22a-1b78df80c185","Type":"ContainerStarted","Data":"955dd8b6f3f0abdb316acdeb0fb39c603ba823cd090a9d71b4bd3f5b36ae98b6"} Oct 30 22:17:27.615839 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:27.615733 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podStartSLOduration=1.615714402 podStartE2EDuration="1.615714402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:27.613195212 +0000 UTC m=+938.870536904" watchObservedRunningTime="2026-10-30 22:17:27.615714402 +0000 UTC m=+938.873056111" Oct 30 22:17:27.841838 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:27.841786 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:28.848388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:28.848350 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-g7jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:28.848840 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:28.848412 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:17:29.845832 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:29.845400 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-g7jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:29.845832 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:29.845461 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:17:30.844228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:30.843613 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-g7jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:30.844228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:30.843677 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:17:31.843229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:31.842645 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-g7jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:31.843229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:31.842723 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:17:32.843652 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:32.843613 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-g7jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:32.844271 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:32.844236 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:17:33.842098 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:33.842057 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-g7jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:33.842341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:33.842122 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:17:34.841797 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:34.841762 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-g7jrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 22:17:34.842345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:34.841824 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 22:17:35.538002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:35.537958 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:17:35.845803 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:35.845733 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:35.846193 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:35.846007 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:35.846630 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:35.846611 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:17:41.654078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.654042 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p"] Oct 30 22:17:41.657924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.657899 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658433 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658479 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658522 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c75851-785f-4799-9e7d-5d2c4510edd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658641 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658570 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658685 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658667 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdl4\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-kube-api-access-scdl4\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658722 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658703 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658723 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658742 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.658854 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.658841 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.754140 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.754104 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p"] Oct 30 22:17:41.759788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.759750 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdl4\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-kube-api-access-scdl4\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.759788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.759807 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.760055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.759839 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.760055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.759867 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.760055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.759913 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.760055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.759980 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.760055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.760010 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.760055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.760034 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c75851-785f-4799-9e7d-5d2c4510edd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.760387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.760075 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.761439 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.761411 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.761612 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.761447 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.762275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.761785 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.762275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.761822 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.762439 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.762217 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c75851-785f-4799-9e7d-5d2c4510edd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.763519 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.763497 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.764399 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.764372 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.818084 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.818049 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.818593 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.818571 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdl4\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-kube-api-access-scdl4\") pod \"maas-default-gateway-openshift-default-75bf867b6c-cv64p\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:41.970988 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:41.970959 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:42.396716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:42.396623 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p"] Oct 30 22:17:42.402862 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:17:42.402829 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c75851_785f_4799_9e7d_5d2c4510edd9.slice/crio-388652c412c4a5a217c5dff9f305d6dee0ce2dc24d484c848184e0544916543a WatchSource:0}: Error finding container 388652c412c4a5a217c5dff9f305d6dee0ce2dc24d484c848184e0544916543a: Status 404 returned error can't find the container with id 388652c412c4a5a217c5dff9f305d6dee0ce2dc24d484c848184e0544916543a Oct 30 22:17:42.405601 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:42.405560 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:42.405687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:42.405652 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:42.405723 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:42.405695 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:17:42.608819 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:42.608786 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" event={"ID":"96c75851-785f-4799-9e7d-5d2c4510edd9","Type":"ContainerStarted","Data":"c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8"} Oct 30 22:17:42.608996 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:42.608846 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" event={"ID":"96c75851-785f-4799-9e7d-5d2c4510edd9","Type":"ContainerStarted","Data":"388652c412c4a5a217c5dff9f305d6dee0ce2dc24d484c848184e0544916543a"} Oct 30 22:17:42.723463 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:42.723409 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" podStartSLOduration=1.723388904 podStartE2EDuration="1.723388904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:42.721933535 +0000 UTC m=+953.979275244" watchObservedRunningTime="2026-10-30 22:17:42.723388904 +0000 UTC m=+953.980730613" Oct 30 22:17:42.971952 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:42.971903 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:43.973495 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:43.973461 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cv64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:17:43.973969 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:43.973515 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:17:44.972242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:44.972185 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cv64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:17:44.972422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:44.972263 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:17:45.971662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:45.971631 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cv64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:17:45.972045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:45.971684 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:17:46.971802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:46.971767 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cv64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:17:46.972263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:46.971854 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:17:47.971770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:47.971735 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cv64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:17:47.971951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:47.971801 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:17:48.971805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:48.971767 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-cv64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 22:17:48.971961 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:48.971830 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 22:17:50.694883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:50.694846 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:50.695306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:50.695231 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:50.695966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:17:50.695947 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:17:54.410814 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:17:54.410777 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:18:02.182448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.182411 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x"] Oct 30 22:18:02.187440 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.187420 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.286549 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.286508 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x"] Oct 30 22:18:02.304692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.304648 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.304890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.304735 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.304890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.304786 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.304890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.304820 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.304890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.304880 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.305144 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.304921 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtqxl\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-kube-api-access-qtqxl\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.305144 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.304986 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2a7ce-218e-4096-a541-f016e8f53f00-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.305144 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.305048 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.305144 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.305077 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406150 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406106 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406162 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtqxl\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-kube-api-access-qtqxl\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406232 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2a7ce-218e-4096-a541-f016e8f53f00-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406285 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406312 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406370 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406673 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406420 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406673 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406471 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406673 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406507 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406673 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406612 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406901 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406716 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.406901 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406736 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.407012 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.406976 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.407067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.407054 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2a7ce-218e-4096-a541-f016e8f53f00-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.408908 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.408880 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.409046 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.408955 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.464490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.464421 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.465089 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.465070 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtqxl\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-kube-api-access-qtqxl\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.500398 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.500366 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:02.924964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.924930 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x"] Oct 30 22:18:02.930270 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:18:02.930239 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c2a7ce_218e_4096_a541_f016e8f53f00.slice/crio-87a30642d7d5023f209f231eb0301c386c21a1447c24dcb95935bc820aeb76e3 WatchSource:0}: Error finding container 87a30642d7d5023f209f231eb0301c386c21a1447c24dcb95935bc820aeb76e3: Status 404 returned error can't find the container with id 87a30642d7d5023f209f231eb0301c386c21a1447c24dcb95935bc820aeb76e3 Oct 30 22:18:02.932629 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.932599 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:02.932706 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.932665 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:02.932706 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:02.932697 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:03.038007 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.037962 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl"] Oct 30 22:18:03.042791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.042760 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.092791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.092755 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-23f3d4-openshift-default-dockercfg-c5fxn\"" Oct 30 22:18:03.102787 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.102755 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl"] Oct 30 22:18:03.114418 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114380 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.114606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114435 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.114606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114552 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfxbd\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-kube-api-access-zfxbd\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.114731 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114606 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.114789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114759 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.114834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114802 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.114873 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114833 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.114918 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114901 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.114953 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.114940 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.215655 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.215559 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.215655 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.215603 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.215655 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.215623 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.215755 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.215804 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.215882 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.215923 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.215969 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfxbd\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-kube-api-access-zfxbd\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.216004 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.216095 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.216235 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.216442 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.216573 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.216689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.216575 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.218479 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.218456 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.218646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.218632 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.273681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.273649 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.273828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.273798 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfxbd\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-kube-api-access-zfxbd\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.358317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.358284 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:03.760122 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.760086 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" event={"ID":"f8c2a7ce-218e-4096-a541-f016e8f53f00","Type":"ContainerStarted","Data":"c30ab5fcef77f84a1bf83e8b63aef01cfad23b0cda7518c8135490115c768d45"} Oct 30 22:18:03.760343 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.760147 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" event={"ID":"f8c2a7ce-218e-4096-a541-f016e8f53f00","Type":"ContainerStarted","Data":"87a30642d7d5023f209f231eb0301c386c21a1447c24dcb95935bc820aeb76e3"} Oct 30 22:18:03.789350 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.789312 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl"] Oct 30 22:18:03.794648 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:18:03.794613 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5678dc0a_2ca6_41ce_852f_064f3566ce4b.slice/crio-d893826905a8d4137beb83c7a6c4bbc8cbf96a98716ee136596cb2a44f3ab1a5 WatchSource:0}: Error finding container d893826905a8d4137beb83c7a6c4bbc8cbf96a98716ee136596cb2a44f3ab1a5: Status 404 returned error can't find the container with id d893826905a8d4137beb83c7a6c4bbc8cbf96a98716ee136596cb2a44f3ab1a5 Oct 30 22:18:03.797378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.797344 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:03.797482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.797413 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:03.797482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.797445 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:18:03.876940 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:03.876879 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podStartSLOduration=1.876861797 podStartE2EDuration="1.876861797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:03.874365362 +0000 UTC m=+975.131707073" watchObservedRunningTime="2026-10-30 22:18:03.876861797 +0000 UTC m=+975.134203511" Oct 30 22:18:04.500831 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:04.500791 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:04.502260 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:04.502236 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:18:04.502339 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:04.502293 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:18:04.780467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:04.780268 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" event={"ID":"5678dc0a-2ca6-41ce-852f-064f3566ce4b","Type":"ContainerStarted","Data":"cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75"} Oct 30 22:18:04.780467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:04.780339 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" event={"ID":"5678dc0a-2ca6-41ce-852f-064f3566ce4b","Type":"ContainerStarted","Data":"d893826905a8d4137beb83c7a6c4bbc8cbf96a98716ee136596cb2a44f3ab1a5"} Oct 30 22:18:04.899133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:04.899070 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podStartSLOduration=2.899051097 podStartE2EDuration="2.899051097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:04.898057415 +0000 UTC m=+976.155399125" watchObservedRunningTime="2026-10-30 22:18:04.899051097 +0000 UTC m=+976.156392805" Oct 30 22:18:05.361440 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:05.361402 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:05.361862 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:05.361835 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:18:05.362039 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:05.362013 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 22:18:05.500947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:05.500910 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:18:05.501361 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:05.500964 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:18:06.364196 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:06.363726 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:18:06.364196 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:06.363795 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 22:18:06.501264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:06.500893 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:18:06.501264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:06.500952 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:18:07.359661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:07.359617 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:18:07.359946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:07.359917 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 22:18:07.501610 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:07.501577 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:18:07.502030 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:07.501640 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:18:08.371232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:08.370651 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:18:08.371232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:08.370728 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 22:18:08.501442 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:08.501256 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:18:08.501442 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:08.501306 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:18:09.361860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:09.361822 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:18:09.362376 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:09.362338 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 22:18:09.500990 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:09.500952 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Oct 30 22:18:09.501142 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:09.501018 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Oct 30 22:18:10.358755 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:10.358715 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Oct 30 22:18:10.358962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:10.358778 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Oct 30 22:18:10.505498 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:10.505465 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:10.505972 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:10.505881 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:10.506648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:10.506629 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:18:11.363808 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:11.363775 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:11.364177 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:11.364155 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:11.364816 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:11.364796 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:18:19.885294 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:19.885263 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:18:24.416301 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:18:24.416251 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:18:26.594926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.594890 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s"] Oct 30 22:18:26.598545 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.598520 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.693811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.693767 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f80a0e\"/\"distinct-w5-f80a0e-kserve-self-signed-certs\"" Oct 30 22:18:26.697891 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.697861 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s"] Oct 30 22:18:26.734779 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.734700 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e362626-cf7a-42c5-8017-2fb7946bd12e-tls-certs\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.735052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.734939 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-tmp-dir\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.735052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.735012 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-dshm\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.735669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.735643 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-model-cache\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.735790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.735760 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc89h\" (UniqueName: \"kubernetes.io/projected/3e362626-cf7a-42c5-8017-2fb7946bd12e-kube-api-access-vc89h\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.735852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.735817 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-home\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.837217 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837167 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-tmp-dir\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.837217 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837222 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-dshm\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.837428 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837335 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-model-cache\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.837465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837448 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc89h\" (UniqueName: \"kubernetes.io/projected/3e362626-cf7a-42c5-8017-2fb7946bd12e-kube-api-access-vc89h\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.837515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837499 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-home\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.837672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837638 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e362626-cf7a-42c5-8017-2fb7946bd12e-tls-certs\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.837672 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837670 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-tmp-dir\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.837930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837901 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-model-cache\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.838052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.837988 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-home\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.839550 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.839523 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-dshm\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.840018 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.839994 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e362626-cf7a-42c5-8017-2fb7946bd12e-tls-certs\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.898036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.897962 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc89h\" (UniqueName: \"kubernetes.io/projected/3e362626-cf7a-42c5-8017-2fb7946bd12e-kube-api-access-vc89h\") pod \"distinct-w5-f80a0e-kserve-7959786b54-m5p9s\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:26.910873 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:26.910827 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:27.293253 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:27.293215 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s"] Oct 30 22:18:27.299541 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:18:27.299501 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e362626_cf7a_42c5_8017_2fb7946bd12e.slice/crio-a8342bcc5eea2410a40387dcf73adb53c205e42af8e05f354f9b890fa4e1180c WatchSource:0}: Error finding container a8342bcc5eea2410a40387dcf73adb53c205e42af8e05f354f9b890fa4e1180c: Status 404 returned error can't find the container with id a8342bcc5eea2410a40387dcf73adb53c205e42af8e05f354f9b890fa4e1180c Oct 30 22:18:27.931064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:27.931031 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" event={"ID":"3e362626-cf7a-42c5-8017-2fb7946bd12e","Type":"ContainerStarted","Data":"f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220"} Oct 30 22:18:27.931485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:27.931075 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" event={"ID":"3e362626-cf7a-42c5-8017-2fb7946bd12e","Type":"ContainerStarted","Data":"a8342bcc5eea2410a40387dcf73adb53c205e42af8e05f354f9b890fa4e1180c"} Oct 30 22:18:28.042282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:28.042226 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podStartSLOduration=2.042188169 podStartE2EDuration="2.042188169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:28.041623445 +0000 UTC m=+999.298965156" watchObservedRunningTime="2026-10-30 22:18:28.042188169 +0000 UTC m=+999.299529879" Oct 30 22:18:28.912694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:28.911400 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:36.911476 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:36.911441 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:36.926429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:36.926398 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:36.991559 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:36.991532 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:18:38.026859 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.026827 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h"] Oct 30 22:18:38.030894 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.030874 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.062257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.062219 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tls-certs\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.062412 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.062331 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tmp-dir\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.062578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.062553 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-model-cache\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.062631 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.062620 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8vc8\" (UniqueName: \"kubernetes.io/projected/4afd7e57-39d4-4adb-8306-8587cdc2a13e-kube-api-access-p8vc8\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.062701 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.062689 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-home\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.062751 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.062711 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-dshm\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.127042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.126995 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"kube-root-ca.crt\"" Oct 30 22:18:38.127042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.127016 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"default-dockercfg-vw7ld\"" Oct 30 22:18:38.127345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.127070 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"distinct-w6-be17e7-kserve-self-signed-certs\"" Oct 30 22:18:38.127345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.127073 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-be17e7\"/\"openshift-service-ca.crt\"" Oct 30 22:18:38.130016 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.129989 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h"] Oct 30 22:18:38.163341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163310 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-home\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.163341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163348 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-dshm\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.163604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163410 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tls-certs\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.163604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163441 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tmp-dir\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.163604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163555 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-model-cache\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.163604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163591 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vc8\" (UniqueName: \"kubernetes.io/projected/4afd7e57-39d4-4adb-8306-8587cdc2a13e-kube-api-access-p8vc8\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.163837 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163768 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-home\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.163908 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163861 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tmp-dir\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.165873 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.163933 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-model-cache\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.166345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.166299 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-dshm\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.166666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.166643 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tls-certs\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.222241 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.222189 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vc8\" (UniqueName: \"kubernetes.io/projected/4afd7e57-39d4-4adb-8306-8587cdc2a13e-kube-api-access-p8vc8\") pod \"distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.343685 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.343585 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:38.727619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.727584 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h"] Oct 30 22:18:38.733481 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:18:38.733455 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4afd7e57_39d4_4adb_8306_8587cdc2a13e.slice/crio-51e63f4b1ccb0fe97034c5d38156823c8ea74ddab3be3bd7db2b1cb5b8ac5dbc WatchSource:0}: Error finding container 51e63f4b1ccb0fe97034c5d38156823c8ea74ddab3be3bd7db2b1cb5b8ac5dbc: Status 404 returned error can't find the container with id 51e63f4b1ccb0fe97034c5d38156823c8ea74ddab3be3bd7db2b1cb5b8ac5dbc Oct 30 22:18:38.992801 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.992715 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" event={"ID":"4afd7e57-39d4-4adb-8306-8587cdc2a13e","Type":"ContainerStarted","Data":"f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519"} Oct 30 22:18:38.992801 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:38.992758 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" event={"ID":"4afd7e57-39d4-4adb-8306-8587cdc2a13e","Type":"ContainerStarted","Data":"51e63f4b1ccb0fe97034c5d38156823c8ea74ddab3be3bd7db2b1cb5b8ac5dbc"} Oct 30 22:18:39.104558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:39.104496 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podStartSLOduration=2.104482915 podStartE2EDuration="2.104482915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:39.102759515 +0000 UTC m=+1010.360101224" watchObservedRunningTime="2026-10-30 22:18:39.104482915 +0000 UTC m=+1010.361824624" Oct 30 22:18:39.344300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:39.344181 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:48.344460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:48.344423 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:48.358161 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:48.358131 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:49.059057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:49.059029 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:18:54.410452 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:18:54.410405 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:18:54.558870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:18:54.558834 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:19:03.725137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.725100 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6"] Oct 30 22:19:03.730615 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.730590 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.767668 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.767635 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.767853 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.767697 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.767853 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.767748 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzvg\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-kube-api-access-qqzvg\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.767853 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.767845 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.767982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.767914 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.767982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.767931 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.768054 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.767990 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.768093 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.768044 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498f6c92-51cd-4ee0-a05d-34d8fa471345-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.768131 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.768117 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.833322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.833289 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6"] Oct 30 22:19:03.869295 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869259 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869491 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869349 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869491 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869374 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzvg\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-kube-api-access-qqzvg\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869491 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869410 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869510 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869536 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869588 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869621 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498f6c92-51cd-4ee0-a05d-34d8fa471345-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869843 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869715 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869843 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869777 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869843 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869799 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.869948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869909 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.870018 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.869994 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.870360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.870341 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498f6c92-51cd-4ee0-a05d-34d8fa471345-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.871848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.871831 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.872015 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.871996 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.930720 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.930688 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:03.943515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:03.943489 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzvg\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-kube-api-access-qqzvg\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:04.043836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:04.043742 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:04.471929 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:04.471893 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6"] Oct 30 22:19:04.478101 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:19:04.478072 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498f6c92_51cd_4ee0_a05d_34d8fa471345.slice/crio-1f4c9b835eb0603fafb6d69ca36ddddd3fec9d537efc0f4c81f14d038f204a6e WatchSource:0}: Error finding container 1f4c9b835eb0603fafb6d69ca36ddddd3fec9d537efc0f4c81f14d038f204a6e: Status 404 returned error can't find the container with id 1f4c9b835eb0603fafb6d69ca36ddddd3fec9d537efc0f4c81f14d038f204a6e Oct 30 22:19:04.480579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:04.480541 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:04.480671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:04.480612 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:04.480671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:04.480642 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:19:05.152759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:05.152722 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" event={"ID":"498f6c92-51cd-4ee0-a05d-34d8fa471345","Type":"ContainerStarted","Data":"2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a"} Oct 30 22:19:05.153409 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:05.152784 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" event={"ID":"498f6c92-51cd-4ee0-a05d-34d8fa471345","Type":"ContainerStarted","Data":"1f4c9b835eb0603fafb6d69ca36ddddd3fec9d537efc0f4c81f14d038f204a6e"} Oct 30 22:19:05.273192 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:05.273134 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podStartSLOduration=2.273119943 podStartE2EDuration="2.273119943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:05.27233852 +0000 UTC m=+1036.529680231" watchObservedRunningTime="2026-10-30 22:19:05.273119943 +0000 UTC m=+1036.530461652" Oct 30 22:19:06.046636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:06.046600 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:06.049078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:06.049045 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:06.049252 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:06.049106 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 22:19:07.048227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:07.047588 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:07.048227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:07.047653 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 22:19:08.048257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:08.044563 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:08.048257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:08.044630 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 22:19:09.047593 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:09.047553 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:09.047886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:09.047855 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 22:19:10.044738 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:10.044696 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:10.045410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:10.045362 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 22:19:11.044584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:11.044540 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 22:19:11.044775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:11.044606 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 22:19:12.049544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:12.049511 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:12.050032 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:12.049913 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:12.050716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:12.050693 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:19:24.412765 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:19:24.412714 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:19:30.997055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:30.997018 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:19:33.576153 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.576119 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q"] Oct 30 22:19:33.581431 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.581409 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.648373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.648331 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-tmp-dir\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.648558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.648405 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdf9t\" (UniqueName: \"kubernetes.io/projected/14a37b06-38fb-43df-9284-8209bb6035a6-kube-api-access-pdf9t\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.648558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.648515 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-home\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.648713 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.648677 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a37b06-38fb-43df-9284-8209bb6035a6-tls-certs\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.648844 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.648765 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-dshm\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.648844 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.648820 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-model-cache\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.677175 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.677136 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eb4aa1\"/\"distinct-2-w2-eb4aa1-kserve-self-signed-certs\"" Oct 30 22:19:33.679618 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.679591 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q"] Oct 30 22:19:33.750530 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.750494 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-tmp-dir\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.750928 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.750889 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdf9t\" (UniqueName: \"kubernetes.io/projected/14a37b06-38fb-43df-9284-8209bb6035a6-kube-api-access-pdf9t\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.752447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.750959 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-home\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.752447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.751025 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-tmp-dir\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.752447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.751058 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a37b06-38fb-43df-9284-8209bb6035a6-tls-certs\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.752447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.751099 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-dshm\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.752447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.751144 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-model-cache\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.752447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.751362 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-home\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.752447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.751525 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-model-cache\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.753923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.753895 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-dshm\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.754337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.754318 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a37b06-38fb-43df-9284-8209bb6035a6-tls-certs\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.809439 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.809404 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdf9t\" (UniqueName: \"kubernetes.io/projected/14a37b06-38fb-43df-9284-8209bb6035a6-kube-api-access-pdf9t\") pod \"distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:33.894663 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:33.894578 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:34.294995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:34.294959 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q"] Oct 30 22:19:34.300972 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:19:34.300942 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14a37b06_38fb_43df_9284_8209bb6035a6.slice/crio-1bafa75878fb3f70101c968cba00b5147c0e2a332c8f197f74d514f4591ac538 WatchSource:0}: Error finding container 1bafa75878fb3f70101c968cba00b5147c0e2a332c8f197f74d514f4591ac538: Status 404 returned error can't find the container with id 1bafa75878fb3f70101c968cba00b5147c0e2a332c8f197f74d514f4591ac538 Oct 30 22:19:34.321922 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:34.321895 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" event={"ID":"14a37b06-38fb-43df-9284-8209bb6035a6","Type":"ContainerStarted","Data":"1bafa75878fb3f70101c968cba00b5147c0e2a332c8f197f74d514f4591ac538"} Oct 30 22:19:35.329711 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:35.329669 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" event={"ID":"14a37b06-38fb-43df-9284-8209bb6035a6","Type":"ContainerStarted","Data":"d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac"} Oct 30 22:19:35.441795 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:35.441739 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podStartSLOduration=2.441724977 podStartE2EDuration="2.441724977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:35.440246027 +0000 UTC m=+1066.697587738" watchObservedRunningTime="2026-10-30 22:19:35.441724977 +0000 UTC m=+1066.699066686" Oct 30 22:19:35.895191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:35.895159 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:43.895536 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:43.895499 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:43.914667 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:43.914634 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:44.398677 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:19:44.398644 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:19:54.411991 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:19:54.411954 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:20:12.682627 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:12.682542 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:20:18.658158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.658122 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9"] Oct 30 22:20:18.662473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.662449 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684172 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684137 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16c9ead-eb60-477e-b415-5a378862ffb5-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684327 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684183 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wzh\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-kube-api-access-b4wzh\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684317 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684346 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684376 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684469 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684485 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684529 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.684665 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.684614 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.711770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.711738 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-eb4aa1-openshift-default-dockercfg-l8phj\"" Oct 30 22:20:18.767712 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.767669 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9"] Oct 30 22:20:18.785769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.785733 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786332 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786300 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786373 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786556 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786498 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786556 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786545 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786569 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786650 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786779 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786699 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786859 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16c9ead-eb60-477e-b415-5a378862ffb5-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786882 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786899 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wzh\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-kube-api-access-b4wzh\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.786947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.786696 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.787880 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.787849 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16c9ead-eb60-477e-b415-5a378862ffb5-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.788099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.788077 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.789408 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.789383 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.790400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.790375 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.844354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.844323 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.844620 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.844597 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wzh\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-kube-api-access-b4wzh\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:18.975266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:18.975231 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:19.402913 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:19.402829 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9"] Oct 30 22:20:19.408751 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:20:19.408724 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16c9ead_eb60_477e_b415_5a378862ffb5.slice/crio-4b4dad667c79e32c1ac64bdcc3a0d7ef3a62933816c9f967e24e571d1a479ea6 WatchSource:0}: Error finding container 4b4dad667c79e32c1ac64bdcc3a0d7ef3a62933816c9f967e24e571d1a479ea6: Status 404 returned error can't find the container with id 4b4dad667c79e32c1ac64bdcc3a0d7ef3a62933816c9f967e24e571d1a479ea6 Oct 30 22:20:19.411337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:19.411299 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:20:19.411414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:19.411389 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:20:19.411449 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:19.411430 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:20:19.586089 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:19.586054 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" event={"ID":"c16c9ead-eb60-477e-b415-5a378862ffb5","Type":"ContainerStarted","Data":"7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74"} Oct 30 22:20:19.586274 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:19.586105 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" event={"ID":"c16c9ead-eb60-477e-b415-5a378862ffb5","Type":"ContainerStarted","Data":"4b4dad667c79e32c1ac64bdcc3a0d7ef3a62933816c9f967e24e571d1a479ea6"} Oct 30 22:20:19.700864 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:19.700815 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" podStartSLOduration=1.7008003999999999 podStartE2EDuration="1.7008004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:19.698455876 +0000 UTC m=+1110.955797586" watchObservedRunningTime="2026-10-30 22:20:19.7008004 +0000 UTC m=+1110.958142110" Oct 30 22:20:19.975771 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:19.975674 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:20.977499 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:20.977460 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:20:20.977917 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:20.977523 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 22:20:21.975888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:21.975853 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:20:21.976058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:21.975911 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 22:20:22.976229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:22.976169 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:20:22.976649 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:22.976267 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 22:20:23.976453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:23.976421 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:20:23.977023 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:23.976488 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 22:20:24.415144 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:20:24.415066 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:20:24.976563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:24.976522 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:20:24.976977 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:24.976585 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 22:20:25.975646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:25.975612 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 22:20:25.975825 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:25.975667 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 22:20:27.029239 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:27.029174 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:27.030039 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:27.030006 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:27.030725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:27.030707 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:20:34.962519 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:34.962482 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6"] Oct 30 22:20:34.967023 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:34.966998 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.016754 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.016728 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d2d4c3-openshift-default-dockercfg-kzbw6\"" Oct 30 22:20:35.072981 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.072946 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6"] Oct 30 22:20:35.097249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097187 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.097444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097296 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tq4\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-kube-api-access-99tq4\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.097444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097342 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.097444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097361 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.097444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097388 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.097444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097444 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.097734 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097470 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.097734 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097492 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.097734 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.097537 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.198802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.198768 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.198825 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tq4\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-kube-api-access-99tq4\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.198867 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.198886 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.198919 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.198991 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199315 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.199024 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199315 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.199059 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199315 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.199095 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.199332 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.199394 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.199427 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.199499 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.199843 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.199825 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.201455 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.201425 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.201665 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.201648 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.258499 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.258429 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tq4\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-kube-api-access-99tq4\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.259228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.259193 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.282347 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.282318 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:35.708847 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.708815 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6"] Oct 30 22:20:35.714849 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:20:35.714808 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd77cda_1cff_4e39_8f54_93097fa58c1c.slice/crio-9075396990de1f5bcc1ea9ea67ebb4d9ebfd2446fc78fca0f6f374d8b3b348f3 WatchSource:0}: Error finding container 9075396990de1f5bcc1ea9ea67ebb4d9ebfd2446fc78fca0f6f374d8b3b348f3: Status 404 returned error can't find the container with id 9075396990de1f5bcc1ea9ea67ebb4d9ebfd2446fc78fca0f6f374d8b3b348f3 Oct 30 22:20:35.717221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.717170 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:20:35.717335 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.717305 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:20:35.717404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:35.717346 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:20:36.691533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:36.691498 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" event={"ID":"0cd77cda-1cff-4e39-8f54-93097fa58c1c","Type":"ContainerStarted","Data":"b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e"} Oct 30 22:20:36.691922 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:36.691549 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" event={"ID":"0cd77cda-1cff-4e39-8f54-93097fa58c1c","Type":"ContainerStarted","Data":"9075396990de1f5bcc1ea9ea67ebb4d9ebfd2446fc78fca0f6f374d8b3b348f3"} Oct 30 22:20:36.856992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:36.856935 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podStartSLOduration=2.856922592 podStartE2EDuration="2.856922592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:36.854627664 +0000 UTC m=+1128.111969375" watchObservedRunningTime="2026-10-30 22:20:36.856922592 +0000 UTC m=+1128.114264302" Oct 30 22:20:37.283411 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:37.283371 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:37.285138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:37.285113 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:20:37.285280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:37.285158 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 22:20:38.283744 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:38.283703 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:20:38.284152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:38.283767 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 22:20:39.283047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:39.283011 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:20:39.283260 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:39.283060 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 22:20:40.283662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:40.283628 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:20:40.284149 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:40.283689 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 22:20:41.283648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:41.283614 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:20:41.283819 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:41.283669 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 22:20:42.282794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:42.282750 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:20:42.283003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:42.282815 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 22:20:43.287558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:43.287525 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:43.288044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:43.287842 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:43.288595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:20:43.288578 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:20:54.412309 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:20:54.412271 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:21:09.038567 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:09.038533 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:21:14.941813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:14.941779 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d"] Oct 30 22:21:14.945999 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:14.945974 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.040769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.040718 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23f3d4\"/\"distinct-2-w3-23f3d4-kserve-self-signed-certs\"" Oct 30 22:21:15.040997 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.040845 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-model-cache\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.040997 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.040917 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-home\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.041158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.041047 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-dshm\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.041158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.041093 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4rh\" (UniqueName: \"kubernetes.io/projected/515e746e-b44c-463a-bc1a-5e57dfb5b920-kube-api-access-bn4rh\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.041299 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.041279 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515e746e-b44c-463a-bc1a-5e57dfb5b920-tls-certs\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.041355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.041340 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-tmp-dir\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.043851 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.043827 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d"] Oct 30 22:21:15.143149 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143101 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515e746e-b44c-463a-bc1a-5e57dfb5b920-tls-certs\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.143400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143168 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-tmp-dir\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.143400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143257 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-model-cache\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.143400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143315 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-home\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.143400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143359 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-dshm\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.143400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143389 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4rh\" (UniqueName: \"kubernetes.io/projected/515e746e-b44c-463a-bc1a-5e57dfb5b920-kube-api-access-bn4rh\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.143674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143638 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-tmp-dir\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.143721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143679 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-home\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.143916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.143895 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-model-cache\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.145764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.145740 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-dshm\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.145933 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.145915 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515e746e-b44c-463a-bc1a-5e57dfb5b920-tls-certs\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.201423 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.201341 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4rh\" (UniqueName: \"kubernetes.io/projected/515e746e-b44c-463a-bc1a-5e57dfb5b920-kube-api-access-bn4rh\") pod \"distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.258395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.258351 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:15.681566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.681528 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d"] Oct 30 22:21:15.688302 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:21:15.688261 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod515e746e_b44c_463a_bc1a_5e57dfb5b920.slice/crio-fe4365ccc20889370a48c260202de2fcdfcb60667fe7f50ed103a71f1c39c056 WatchSource:0}: Error finding container fe4365ccc20889370a48c260202de2fcdfcb60667fe7f50ed103a71f1c39c056: Status 404 returned error can't find the container with id fe4365ccc20889370a48c260202de2fcdfcb60667fe7f50ed103a71f1c39c056 Oct 30 22:21:15.952221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.952119 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" event={"ID":"515e746e-b44c-463a-bc1a-5e57dfb5b920","Type":"ContainerStarted","Data":"0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3"} Oct 30 22:21:15.952221 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:15.952166 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" event={"ID":"515e746e-b44c-463a-bc1a-5e57dfb5b920","Type":"ContainerStarted","Data":"fe4365ccc20889370a48c260202de2fcdfcb60667fe7f50ed103a71f1c39c056"} Oct 30 22:21:16.063426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:16.063371 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podStartSLOduration=2.063358039 podStartE2EDuration="2.063358039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:16.061742043 +0000 UTC m=+1167.319083764" watchObservedRunningTime="2026-10-30 22:21:16.063358039 +0000 UTC m=+1167.320699767" Oct 30 22:21:16.258715 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:16.258618 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:24.413577 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:21:24.413537 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:21:25.258587 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:25.258548 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:25.271153 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:25.271126 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:26.024160 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:26.024133 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:21:34.810900 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.810854 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8"] Oct 30 22:21:34.815467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.815442 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885342 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885300 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885534 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885484 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885534 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885509 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885626 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885603 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885648 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f6c12a-6677-4996-a7b9-fa3c581e7873-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885705 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885667 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885766 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885742 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885811 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.885861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.885831 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp57g\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-kube-api-access-xp57g\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.918484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.918450 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8"] Oct 30 22:21:34.987358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987315 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987357 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987625 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987456 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987693 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f6c12a-6677-4996-a7b9-fa3c581e7873-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987741 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987802 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987834 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987838 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-data\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987859 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp57g\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-kube-api-access-xp57g\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.987982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.987904 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.988425 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.988399 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f6c12a-6677-4996-a7b9-fa3c581e7873-istiod-ca-cert\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.988425 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.988409 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-credential-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.988563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.988544 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-socket\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.988651 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.988618 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-certs\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.990470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.990442 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-podinfo\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:34.990557 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:34.990451 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-envoy\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:35.047706 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.047670 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-token\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:35.047939 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.047914 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp57g\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-kube-api-access-xp57g\") pod \"e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:35.130409 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.130311 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:35.133598 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.133565 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs"] Oct 30 22:21:35.138352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.138318 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.187395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.187352 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-be17e7-openshift-default-dockercfg-bb9z8\"" Oct 30 22:21:35.191126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.190862 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7vs\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-kube-api-access-fb7vs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.191126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.190921 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.191126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.190976 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.191126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.191021 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.191126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.191049 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.191126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.191072 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.191126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.191107 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.191617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.191254 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dfee67-2e63-498b-ad2c-efbdd34146ae-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.191617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.191283 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.200935 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.200884 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs"] Oct 30 22:21:35.292656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.292618 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7vs\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-kube-api-access-fb7vs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.292863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.292666 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.292863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.292703 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.292863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.292738 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.292863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.292757 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.292863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.292832 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.293125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.292880 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.293125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.293025 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dfee67-2e63-498b-ad2c-efbdd34146ae-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.293125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.293052 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.293300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.293258 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.293341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.293293 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.293436 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.293412 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.293488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.293440 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.293838 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.293814 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dfee67-2e63-498b-ad2c-efbdd34146ae-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.295116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.295094 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.295349 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.295334 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.351091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.351054 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.351266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.351172 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7vs\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-kube-api-access-fb7vs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.466993 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.466960 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:35.573132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.573097 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8"] Oct 30 22:21:35.579024 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:21:35.578997 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f6c12a_6677_4996_a7b9_fa3c581e7873.slice/crio-a56448d51b0e8ea6439aaf28b91a0873bb498eb663b9d7e799c7674e6add3cf2 WatchSource:0}: Error finding container a56448d51b0e8ea6439aaf28b91a0873bb498eb663b9d7e799c7674e6add3cf2: Status 404 returned error can't find the container with id a56448d51b0e8ea6439aaf28b91a0873bb498eb663b9d7e799c7674e6add3cf2 Oct 30 22:21:35.582526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.582489 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:35.582644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.582556 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:35.582644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.582588 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:35.854732 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.854651 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs"] Oct 30 22:21:35.861654 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:21:35.861624 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16dfee67_2e63_498b_ad2c_efbdd34146ae.slice/crio-959dac9a1af7eca194f2dbbba053b2553e095e850bf3984e7f6647d6b4ff42ce WatchSource:0}: Error finding container 959dac9a1af7eca194f2dbbba053b2553e095e850bf3984e7f6647d6b4ff42ce: Status 404 returned error can't find the container with id 959dac9a1af7eca194f2dbbba053b2553e095e850bf3984e7f6647d6b4ff42ce Oct 30 22:21:35.865301 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.865263 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:35.865397 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.865358 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:35.865436 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:35.865398 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:21:36.077729 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.077690 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" event={"ID":"42f6c12a-6677-4996-a7b9-fa3c581e7873","Type":"ContainerStarted","Data":"ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808"} Oct 30 22:21:36.077920 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.077751 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" event={"ID":"42f6c12a-6677-4996-a7b9-fa3c581e7873","Type":"ContainerStarted","Data":"a56448d51b0e8ea6439aaf28b91a0873bb498eb663b9d7e799c7674e6add3cf2"} Oct 30 22:21:36.079517 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.079490 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" event={"ID":"16dfee67-2e63-498b-ad2c-efbdd34146ae","Type":"ContainerStarted","Data":"1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61"} Oct 30 22:21:36.079640 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.079540 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" event={"ID":"16dfee67-2e63-498b-ad2c-efbdd34146ae","Type":"ContainerStarted","Data":"959dac9a1af7eca194f2dbbba053b2553e095e850bf3984e7f6647d6b4ff42ce"} Oct 30 22:21:36.131023 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.130933 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:36.132596 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.132562 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:21:36.132666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.132630 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 22:21:36.191758 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.191703 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podStartSLOduration=2.191688155 podStartE2EDuration="2.191688155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:36.190565635 +0000 UTC m=+1187.447907344" watchObservedRunningTime="2026-10-30 22:21:36.191688155 +0000 UTC m=+1187.449029866" Oct 30 22:21:36.306283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.306221 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podStartSLOduration=1.306187751 podStartE2EDuration="1.306187751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:36.303276338 +0000 UTC m=+1187.560618049" watchObservedRunningTime="2026-10-30 22:21:36.306187751 +0000 UTC m=+1187.563529461" Oct 30 22:21:36.468013 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.467973 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:36.469961 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.469934 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:21:36.470091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:36.469985 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 22:21:37.136228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:37.134815 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:21:37.136228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:37.134880 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 22:21:37.484257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:37.483540 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:21:37.484257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:37.483609 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 22:21:38.132231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:38.131850 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:21:38.132231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:38.131937 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 22:21:38.473425 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:38.473385 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:21:38.474065 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:38.474030 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 22:21:39.132693 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:39.132648 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:21:39.132995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:39.132962 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 22:21:39.471351 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:39.471276 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:21:39.471643 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:39.471614 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 22:21:40.143888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:40.143849 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:21:40.144500 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:40.144467 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 22:21:40.469716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:40.469675 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:21:40.470003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:40.469974 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 22:21:41.134861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.134821 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:21:41.135648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.135614 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 22:21:41.305120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.305076 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 22:21:41.328229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.321299 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 22:21:41.384229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.383407 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.407289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.390288 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.475230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.474385 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:21:41.475230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.474447 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 22:21:41.500230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.492309 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.522228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.519070 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.562228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.561326 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:21:41.562228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.561403 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.755229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.754416 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:41.880229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:41.879345 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 22:21:42.001494 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.001445 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 22:21:42.021516 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.021374 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:42.071840 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.059598 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 22:21:42.194802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.194759 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 22:21:42.220327 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.195397 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:21:42.220327 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.195448 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 22:21:42.442534 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.442489 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:42.443785 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.443763 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:21:42.570712 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.570665 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:21:42.601481 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.601448 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:21:42.604428 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.601849 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:21:42.604675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.604650 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 22:21:42.619232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.607690 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:21:42.654325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.654292 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:21:42.693274 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.693035 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:21:42.704909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.704675 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:21:42.715059 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:42.715030 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:21:43.132249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:43.132130 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 22:21:43.132249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:43.132233 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 22:21:43.468216 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:43.468155 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 22:21:43.468637 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:43.468265 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 22:21:44.142463 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:44.142426 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:44.143017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:44.142989 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:44.143960 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:44.143934 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:21:44.471971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:44.471942 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:44.472473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:44.472324 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:44.473547 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:44.473521 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:21:49.070230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:49.070180 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:21:49.070790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:21:49.070629 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:21:54.412217 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:21:54.412164 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:22:03.769570 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:03.769532 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:22:04.003431 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.003387 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp"] Oct 30 22:22:04.008863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.008838 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.097705 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.097607 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp"] Oct 30 22:22:04.098566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.098535 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.098702 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.098667 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.098769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.098752 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxml\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-kube-api-access-mnxml\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.098889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.098796 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.098889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.098828 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.099020 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.099002 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.099191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.099159 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.099316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.099228 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.099316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.099301 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.143877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.143841 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq"] Oct 30 22:22:04.148262 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.148236 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.197736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.197709 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b13755-openshift-default-dockercfg-45x5j\"" Oct 30 22:22:04.200380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200356 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200474 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200446 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200478 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200495 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200535 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200564 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200587 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxml\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-kube-api-access-mnxml\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200613 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200642 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.200915 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.200889 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.201267 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.201237 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-credential-socket\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.201267 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.201256 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-data\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.201482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.201289 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istiod-ca-cert\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.201482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.201334 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-certs\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.202862 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.202839 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-envoy\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.203186 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.203168 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-podinfo\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.213688 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.213658 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq"] Oct 30 22:22:04.259266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.259228 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-token\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.265643 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.263264 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxml\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-kube-api-access-mnxml\") pod \"e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.315495 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.309993 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72337f40-32a4-4136-b4e4-4b8e49399673-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.328225 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.328180 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85kbf\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-kube-api-access-85kbf\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.329795 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.329762 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.330097 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.324095 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:04.330245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.330074 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.330493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.330472 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.330641 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.330623 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.330725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.330665 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.330784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.330768 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72337f40-32a4-4136-b4e4-4b8e49399673-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.330854 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.330831 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.432898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.431919 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.432898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.431971 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.432898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.432088 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.432898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.432132 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.432898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.432164 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.432898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.432682 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.433384 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.432972 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.433384 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.433254 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.433538 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.433516 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.440031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.439999 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72337f40-32a4-4136-b4e4-4b8e49399673-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.440238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.440069 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.440238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.440177 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72337f40-32a4-4136-b4e4-4b8e49399673-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.440238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.440230 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85kbf\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-kube-api-access-85kbf\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.443004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.442935 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72337f40-32a4-4136-b4e4-4b8e49399673-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.443521 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.443497 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.443976 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.443957 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72337f40-32a4-4136-b4e4-4b8e49399673-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.530810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.529092 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85kbf\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-kube-api-access-85kbf\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.590426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.588513 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.772393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.772347 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:04.783437 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.783409 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp"] Oct 30 22:22:04.788626 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:22:04.788577 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb179ec_ed21_46d3_88e4_45185d9e22d7.slice/crio-12571f37df2725da6c42a21500ba98fe31f4e4655fd54ac04d48e86dc95e4208 WatchSource:0}: Error finding container 12571f37df2725da6c42a21500ba98fe31f4e4655fd54ac04d48e86dc95e4208: Status 404 returned error can't find the container with id 12571f37df2725da6c42a21500ba98fe31f4e4655fd54ac04d48e86dc95e4208 Oct 30 22:22:04.791964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.791914 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:04.792099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.792022 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:04.792099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.792069 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:04.820736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.820705 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m"] Oct 30 22:22:04.829109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.829068 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884066 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7054201d-3c4a-4839-b282-295d853eb43c-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884175 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6gf8\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-kube-api-access-j6gf8\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884267 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884327 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884371 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884397 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7054201d-3c4a-4839-b282-295d853eb43c-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884435 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884487 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:04.885085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:04.884512 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.044620 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.044699 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.044749 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.044773 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7054201d-3c4a-4839-b282-295d853eb43c-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.044808 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.044856 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.044881 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.045004 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7054201d-3c4a-4839-b282-295d853eb43c-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.045113 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6gf8\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-kube-api-access-j6gf8\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.045889 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.046169 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.046450 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.063695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.047100 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7054201d-3c4a-4839-b282-295d853eb43c-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.082902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.082819 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.132451 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.126272 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m"] Oct 30 22:22:05.256256 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.231999 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6gf8\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-kube-api-access-j6gf8\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.256909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.256825 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.257300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.257274 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7054201d-3c4a-4839-b282-295d853eb43c-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.359315 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.351876 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.469819 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.469488 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:05.519964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.519928 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq"] Oct 30 22:22:05.569900 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:22:05.569839 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72337f40_32a4_4136_b4e4_4b8e49399673.slice/crio-bf2a9d78508d1b8557164b78dcd06581dcfd2d3a03e3e7c3ebfe2ea318bf0234 WatchSource:0}: Error finding container bf2a9d78508d1b8557164b78dcd06581dcfd2d3a03e3e7c3ebfe2ea318bf0234: Status 404 returned error can't find the container with id bf2a9d78508d1b8557164b78dcd06581dcfd2d3a03e3e7c3ebfe2ea318bf0234 Oct 30 22:22:05.596073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.595745 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:05.596073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.595852 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:05.596073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.595899 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:05.978227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.973136 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" event={"ID":"3eb179ec-ed21-46d3-88e4-45185d9e22d7","Type":"ContainerStarted","Data":"4448910437af5e2b64a165cce513a1145a797506213b043bd7700ff2a1653642"} Oct 30 22:22:05.978227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.973225 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" event={"ID":"3eb179ec-ed21-46d3-88e4-45185d9e22d7","Type":"ContainerStarted","Data":"12571f37df2725da6c42a21500ba98fe31f4e4655fd54ac04d48e86dc95e4208"} Oct 30 22:22:05.997470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:05.995807 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" event={"ID":"72337f40-32a4-4136-b4e4-4b8e49399673","Type":"ContainerStarted","Data":"bf2a9d78508d1b8557164b78dcd06581dcfd2d3a03e3e7c3ebfe2ea318bf0234"} Oct 30 22:22:06.130729 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.130299 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podStartSLOduration=3.130264274 podStartE2EDuration="3.130264274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:06.106325581 +0000 UTC m=+1217.363667293" watchObservedRunningTime="2026-10-30 22:22:06.130264274 +0000 UTC m=+1217.387605987" Oct 30 22:22:06.482232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.335471 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:06.482232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.337304 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 22:22:06.482232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.392037 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:06.821298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.736748 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:06.821298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.736830 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 22:22:06.821298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.741892 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 22:22:06.821298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.743793 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 22:22:06.821298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.756851 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded" Oct 30 22:22:06.866088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.845543 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 22:22:06.906311 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.906266 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 22:22:07.105948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:06.999759 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 22:22:07.105948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.039363 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:07.108960 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.108918 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:22:07.114219 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.112433 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 22:22:07.149310 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.135649 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:07.178130 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.167225 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:07.178130 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.171356 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:07.290577 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.290521 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.321569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.321528 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.321820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.321798 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:07.328698 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.328655 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:07.328933 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.328910 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 22:22:07.340074 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.340035 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:07.341682 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.341636 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded" Oct 30 22:22:07.356446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.351221 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:22:07.356446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.351269 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m"] Oct 30 22:22:07.508394 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.508337 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:07.547102 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.544835 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:22:07.608792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.608670 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" event={"ID":"72337f40-32a4-4136-b4e4-4b8e49399673","Type":"ContainerStarted","Data":"185eb5c36354a4c01fb796019c15ec7d43448169b1224eb709c692597d2d6c7c"} Oct 30 22:22:07.697369 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.697341 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:22:07.749862 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.749800 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:07.750822 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.750478 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:22:07.827772 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.803026 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:07.845348 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.845313 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:07.845779 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.845746 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:22:07.894468 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.890893 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podStartSLOduration=3.890869141 podStartE2EDuration="3.890869141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:07.868691205 +0000 UTC m=+1219.126032929" watchObservedRunningTime="2026-10-30 22:22:07.890869141 +0000 UTC m=+1219.148210856" Oct 30 22:22:07.902829 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.902192 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:22:07.921413 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:07.914746 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:22:08.324587 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:08.324542 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:08.332328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:08.332283 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 22:22:08.618544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:08.618448 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" event={"ID":"7054201d-3c4a-4839-b282-295d853eb43c","Type":"ContainerStarted","Data":"fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e"} Oct 30 22:22:08.618544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:08.618512 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" event={"ID":"7054201d-3c4a-4839-b282-295d853eb43c","Type":"ContainerStarted","Data":"44747f333aa443e161480f4bbe949b8bbedb41240fd7ba5c30248f5508e13abd"} Oct 30 22:22:08.774228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:08.773492 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:08.774228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:08.773576 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:22:09.327696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:09.327653 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:09.328302 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:09.328267 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 22:22:09.476309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:09.476273 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:09.476592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:09.476553 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:09.476760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:09.476727 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:22:09.776231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:09.775622 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:09.776231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:09.775684 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:22:10.329426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:10.327603 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:10.329426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:10.327665 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 22:22:10.475229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:10.474642 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:10.475229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:10.474704 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:22:10.773651 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:10.773611 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:10.773957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:10.773924 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:22:11.332230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:11.327616 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:11.332230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:11.327680 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 22:22:11.474154 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:11.473617 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:11.474154 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:11.473699 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:22:11.792232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:11.791622 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:11.792232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:11.791689 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:22:12.327241 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:12.326687 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:12.327241 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:12.326748 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 22:22:12.478231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:12.471769 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:12.478231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:12.471834 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:22:12.782228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:12.779706 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:12.782228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:12.779787 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:22:13.331232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:13.327677 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 22:22:13.331232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:13.327749 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 22:22:13.474233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:13.473763 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:13.474233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:13.473839 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:22:13.775619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:13.775586 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 22:22:13.776015 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:13.775639 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 22:22:14.378176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:14.378139 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:14.379632 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:14.379606 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:14.381987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:14.381966 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:22:14.472277 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:14.472188 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:14.472460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:14.472317 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:22:14.579019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:14.578951 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podStartSLOduration=10.578936807 podStartE2EDuration="10.578936807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:08.741723975 +0000 UTC m=+1219.999065686" watchObservedRunningTime="2026-10-30 22:22:14.578936807 +0000 UTC m=+1225.836278516" Oct 30 22:22:15.189988 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:15.189952 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:15.191199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:15.191165 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:15.192641 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:15.192617 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:22:15.472509 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:15.472417 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 22:22:15.472509 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:15.472472 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 22:22:16.476911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:16.476878 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:16.477341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:16.477163 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:16.477985 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:16.477963 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:22:19.586382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.586347 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd"] Oct 30 22:22:19.590861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.590837 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.694132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.694083 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd"] Oct 30 22:22:19.717805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.717764 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.717805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.717819 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.718088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.717895 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.718088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.718048 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.718252 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.718170 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.718328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.718275 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.718388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.718334 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.718445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.718390 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.718505 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.718450 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfz7\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-kube-api-access-gnfz7\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820283 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820328 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820349 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820444 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820545 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820610 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820671 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820712 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820769 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfz7\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-kube-api-access-gnfz7\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820802 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.820924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.820844 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.821083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.821012 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.821452 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.821425 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.821963 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.821941 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.823658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.823632 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.824147 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.824125 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.880646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.880550 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfz7\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-kube-api-access-gnfz7\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.880972 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.880948 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:19.910449 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:19.910410 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:20.340488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:20.340448 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd"] Oct 30 22:22:20.346830 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:22:20.346799 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09f3234_84e4_4ff0_b9cf_6b9e4205cec6.slice/crio-30e51a5a7d969ab3a2c7fb847534b73a78a35f8e689c5542336ec91511849d45 WatchSource:0}: Error finding container 30e51a5a7d969ab3a2c7fb847534b73a78a35f8e689c5542336ec91511849d45: Status 404 returned error can't find the container with id 30e51a5a7d969ab3a2c7fb847534b73a78a35f8e689c5542336ec91511849d45 Oct 30 22:22:20.349566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:20.349529 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.349675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:20.349602 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.349675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:20.349629 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:20.790300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:20.790260 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" event={"ID":"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6","Type":"ContainerStarted","Data":"4be0e230e4e8caac589fc980efc0c81c3ab541be8c7be2227fba745f3e867cd9"} Oct 30 22:22:20.790800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:20.790321 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" event={"ID":"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6","Type":"ContainerStarted","Data":"30e51a5a7d969ab3a2c7fb847534b73a78a35f8e689c5542336ec91511849d45"} Oct 30 22:22:20.906574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:20.906511 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podStartSLOduration=1.906492524 podStartE2EDuration="1.906492524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:20.905253326 +0000 UTC m=+1232.162595036" watchObservedRunningTime="2026-10-30 22:22:20.906492524 +0000 UTC m=+1232.163834235" Oct 30 22:22:20.911293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:20.911248 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:21.913016 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:21.912983 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:21.913465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:21.913035 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 22:22:22.910866 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:22.910834 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:22.911046 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:22.910891 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 22:22:23.910925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:23.910879 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:23.911457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:23.910952 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 22:22:24.414025 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:22:24.413982 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:22:24.911244 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:24.911212 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:24.911646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:24.911274 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 22:22:25.911021 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:25.910984 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:25.911183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:25.911039 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 22:22:26.911613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:26.911570 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 22:22:26.912109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:26.911632 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 22:22:27.916664 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:27.916631 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:27.917120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:27.916997 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:27.917723 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:27.917696 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:22:35.609559 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.609522 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4"] Oct 30 22:22:35.613932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.613909 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758185 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758151 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758232 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758347 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b313574-95b0-4a9d-80f3-37c68537b527-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758464 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758392 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b313574-95b0-4a9d-80f3-37c68537b527-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758510 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758502 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758547 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758522 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758593 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758544 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-kube-api-access-7jzml\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758667 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758653 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.758703 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.758692 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.819560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.819517 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4"] Oct 30 22:22:35.860074 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.859974 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860074 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860051 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860146 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860236 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860285 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b313574-95b0-4a9d-80f3-37c68537b527-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860320 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b313574-95b0-4a9d-80f3-37c68537b527-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860518 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860453 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860518 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860482 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860618 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860519 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-kube-api-access-7jzml\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.860857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.860829 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-data\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.861247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.861186 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-credential-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.861429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.861406 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-socket\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.861583 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.861559 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-certs\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.861691 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.861674 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b313574-95b0-4a9d-80f3-37c68537b527-istiod-ca-cert\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.862848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.862828 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-envoy\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.864588 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.864554 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b313574-95b0-4a9d-80f3-37c68537b527-istio-podinfo\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.886811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.886781 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq"] Oct 30 22:22:35.890800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.890778 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.917293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.917254 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-istio-token\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.917519 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.917499 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-kube-api-access-7jzml\") pod \"e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.928803 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.928777 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:35.961750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.961712 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.961931 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.961756 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.961931 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.961804 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.962071 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.961940 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-959x7\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-kube-api-access-959x7\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.962071 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.962030 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.962177 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.962117 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.962177 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.962158 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.962325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.962189 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.962325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.962318 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:35.979272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:35.979235 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq"] Oct 30 22:22:36.063963 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.063928 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064170 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.063975 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064170 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.063995 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064170 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.064021 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064170 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.064102 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.064268 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.064293 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.064341 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.064417 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-959x7\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-kube-api-access-959x7\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.064865 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.064838 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.065250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.065030 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.065250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.065066 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.065616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.065435 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.065616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.065581 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.067161 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.067122 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.067713 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.067690 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.121640 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.121566 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.121776 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.121636 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-959x7\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-kube-api-access-959x7\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.205355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.205321 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:36.362743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.362699 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4"] Oct 30 22:22:36.368141 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:22:36.368112 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b313574_95b0_4a9d_80f3_37c68537b527.slice/crio-69979b4e65e4a5637aaf54e6fbb66671c657624149beb40beb886a4e856f056f WatchSource:0}: Error finding container 69979b4e65e4a5637aaf54e6fbb66671c657624149beb40beb886a4e856f056f: Status 404 returned error can't find the container with id 69979b4e65e4a5637aaf54e6fbb66671c657624149beb40beb886a4e856f056f Oct 30 22:22:36.370758 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.370721 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.370849 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.370789 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.370849 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.370819 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.583956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.583916 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq"] Oct 30 22:22:36.591498 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:22:36.591460 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b61aa24_c615_44fd_a2ae_fd7f075ede95.slice/crio-8fac4d850a2f6999705c239a07fde85344ccf2213acc52c843eaf0c623666c14 WatchSource:0}: Error finding container 8fac4d850a2f6999705c239a07fde85344ccf2213acc52c843eaf0c623666c14: Status 404 returned error can't find the container with id 8fac4d850a2f6999705c239a07fde85344ccf2213acc52c843eaf0c623666c14 Oct 30 22:22:36.594738 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.594695 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.594846 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.594796 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.594846 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.594839 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:22:36.895242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.895121 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" event={"ID":"8b61aa24-c615-44fd-a2ae-fd7f075ede95","Type":"ContainerStarted","Data":"6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7"} Oct 30 22:22:36.895242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.895183 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" event={"ID":"8b61aa24-c615-44fd-a2ae-fd7f075ede95","Type":"ContainerStarted","Data":"8fac4d850a2f6999705c239a07fde85344ccf2213acc52c843eaf0c623666c14"} Oct 30 22:22:36.897011 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.896978 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" event={"ID":"6b313574-95b0-4a9d-80f3-37c68537b527","Type":"ContainerStarted","Data":"cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2"} Oct 30 22:22:36.897124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.897038 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" event={"ID":"6b313574-95b0-4a9d-80f3-37c68537b527","Type":"ContainerStarted","Data":"69979b4e65e4a5637aaf54e6fbb66671c657624149beb40beb886a4e856f056f"} Oct 30 22:22:36.929836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:36.929801 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:37.013662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:37.013602 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podStartSLOduration=2.013585525 podStartE2EDuration="2.013585525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:37.013070592 +0000 UTC m=+1248.270412302" watchObservedRunningTime="2026-10-30 22:22:37.013585525 +0000 UTC m=+1248.270927234" Oct 30 22:22:37.146778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:37.146669 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" podStartSLOduration=2.146654093 podStartE2EDuration="2.146654093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:37.14587269 +0000 UTC m=+1248.403214401" watchObservedRunningTime="2026-10-30 22:22:37.146654093 +0000 UTC m=+1248.403995825" Oct 30 22:22:37.205829 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:37.205787 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:37.207664 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:37.207637 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:22:37.207810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:37.207685 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 22:22:37.952230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:37.951333 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:37.952230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:37.951398 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 22:22:38.205732 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:38.205652 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:22:38.205946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:38.205920 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 22:22:38.950232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:38.946273 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:38.950232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:38.946337 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 22:22:39.206484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:39.206394 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:22:39.206484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:39.206454 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 22:22:39.967230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:39.963515 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:39.967230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:39.963595 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 22:22:40.206254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:40.206059 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:22:40.206254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:40.206142 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 22:22:40.944229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:40.934713 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:40.944229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:40.934790 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 22:22:41.205770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:41.205677 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:22:41.205770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:41.205746 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 22:22:41.933054 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:41.932605 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:41.933054 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:41.932668 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 22:22:42.206472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:42.206386 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:22:42.206472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:42.206438 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 22:22:42.931233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:42.929556 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 22:22:42.931233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:42.929629 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 22:22:43.206457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:43.206361 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:22:43.206457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:43.206411 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 22:22:44.131090 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:44.131047 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:44.131489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:44.131468 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:44.132625 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:44.132601 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:22:44.575882 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:44.575849 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:44.576403 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:44.576312 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:44.577245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:44.577191 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:22:48.877232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:48.876938 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:48.916228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:48.915233 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:48.986230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:48.985751 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.030655 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.030216 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.140282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.136696 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.741235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.703456 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.741235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.704257 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.741235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.704343 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.741235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.724602 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.833156 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.832365 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.922247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.910754 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.926225 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.923928 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.926225 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.924055 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:22:49.926225 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.924838 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 22:22:49.926225 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.925528 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:22:49.991305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.989297 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:49.991305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:49.989702 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:22:50.036239 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.033312 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.036239 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.034522 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:22:50.076290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.057325 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.197441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.197400 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.198098 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.198072 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:22:50.714738 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.714696 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 22:22:50.726680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.726637 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:22:50.727176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.727150 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 22:22:50.727634 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.727613 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:22:50.727801 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.727787 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:22:50.727936 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.727923 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:22:50.745235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.730456 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.745235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.730693 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:22:50.905229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.887692 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 22:22:50.905229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.888136 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:22:50.925641 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.925591 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:50.926426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.926396 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 22:22:50.986494 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.933314 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.021230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:50.999583 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 22:22:51.051547 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.050306 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.074035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.073975 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.219454 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.217349 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 22:22:51.307268 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.305795 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:22:51.307268 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.305896 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:22:51.307268 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.305955 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:22:51.307268 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.306000 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:22:51.767342 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.765286 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.769764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.769742 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:22:51.779288 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.774515 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:22:51.791765 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.774544 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 22:22:51.812872 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.812846 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:22:51.846841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.843920 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:22:51.872891 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.849862 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:22:51.872891 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.852574 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:22:51.872891 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.855476 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:22:51.892724 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.887910 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:22:51.892724 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.888964 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:22:51.892724 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.890442 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:22:51.923554 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.923516 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:22:51.927975 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:22:51.927950 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:22:54.411315 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:22:54.411278 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:23:12.539736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:12.539656 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p"] Oct 30 22:23:12.540333 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:12.540009 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" containerID="cri-o://c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8" gracePeriod=30 Oct 30 22:23:12.586338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:12.586301 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj"] Oct 30 22:23:12.586582 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:12.586556 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" containerID="cri-o://5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9" gracePeriod=30 Oct 30 22:23:17.867959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.867926 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:23:17.924158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.924133 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:23:17.935009 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.934984 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-token\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-token\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935020 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c75851-785f-4799-9e7d-5d2c4510edd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c75851-785f-4799-9e7d-5d2c4510edd9-istiod-ca-cert\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935252 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935126 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-data\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935252 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935238 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-credential-socket\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935369 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935345 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-socket\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935425 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935391 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-certs\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935483 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935456 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-kube-api-access-scdl4\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-kube-api-access-scdl4\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935494 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-podinfo\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935420 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-data" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:17.935542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935535 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-envoy\") pod \"96c75851-785f-4799-9e7d-5d2c4510edd9\" (UID: \"96c75851-785f-4799-9e7d-5d2c4510edd9\") " Oct 30 22:23:17.935542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935510 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c75851-785f-4799-9e7d-5d2c4510edd9-istiod-ca-cert" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:17.935735 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935600 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-socket" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:17.935735 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935637 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-certs" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:17.935843 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.935800 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-credential-socket" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:17.936248 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.936192 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:17.936364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.936255 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:17.936364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.936274 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:17.936364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.936290 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:17.936364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.936303 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c75851-785f-4799-9e7d-5d2c4510edd9-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:17.937448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.937412 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-token" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:17.938613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.938586 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-podinfo" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:17.938613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.938592 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-kube-api-access-scdl4" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "kube-api-access-scdl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:17.938741 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:17.938634 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-envoy" pod "96c75851-785f-4799-9e7d-5d2c4510edd9" (UID: "96c75851-785f-4799-9e7d-5d2c4510edd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:18.037099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037066 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-credential-socket\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.037303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037190 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-podinfo\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.037303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037263 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fdc873-78c7-404f-b22a-1b78df80c185-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fdc873-78c7-404f-b22a-1b78df80c185-istiod-ca-cert\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.037425 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037407 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-socket\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.037484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037409 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-credential-socket" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:18.037484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037448 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-token\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-token\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.037585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037482 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-kube-api-access-qws6l\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-kube-api-access-qws6l\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.037585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037557 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-envoy\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.037689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037603 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-data\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.037689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037633 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-socket" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:18.037689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.037681 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-certs\") pod \"c1fdc873-78c7-404f-b22a-1b78df80c185\" (UID: \"c1fdc873-78c7-404f-b22a-1b78df80c185\") " Oct 30 22:23:18.038030 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.038001 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fdc873-78c7-404f-b22a-1b78df80c185-istiod-ca-cert" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:18.038475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.038449 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.038579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.038478 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.038579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.038493 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.038579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.038507 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scdl4\" (UniqueName: \"kubernetes.io/projected/96c75851-785f-4799-9e7d-5d2c4510edd9-kube-api-access-scdl4\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.038579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.038521 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.038579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.038535 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c75851-785f-4799-9e7d-5d2c4510edd9-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.038579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.038547 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fdc873-78c7-404f-b22a-1b78df80c185-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.039422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.039397 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-certs" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:18.040031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.040001 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-podinfo" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:18.040129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.040073 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-envoy" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:18.040232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.040196 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-data" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:18.040524 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.040499 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-token" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:18.040795 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.040774 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-kube-api-access-qws6l" pod "c1fdc873-78c7-404f-b22a-1b78df80c185" (UID: "c1fdc873-78c7-404f-b22a-1b78df80c185"). InnerVolumeSpecName "kube-api-access-qws6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:18.140719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.140645 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.140719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.140674 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qws6l\" (UniqueName: \"kubernetes.io/projected/c1fdc873-78c7-404f-b22a-1b78df80c185-kube-api-access-qws6l\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.140719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.140685 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.140719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.140694 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.140719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.140703 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fdc873-78c7-404f-b22a-1b78df80c185-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.140719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.140714 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fdc873-78c7-404f-b22a-1b78df80c185-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:18.446376 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.446339 5226 generic.go:356] "Generic (PLEG): container finished" podID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerID="c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8" exitCode=0 Oct 30 22:23:18.446543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.446393 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" Oct 30 22:23:18.446543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.446396 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" event={"ID":"96c75851-785f-4799-9e7d-5d2c4510edd9","Type":"ContainerDied","Data":"c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8"} Oct 30 22:23:18.446543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.446457 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p" event={"ID":"96c75851-785f-4799-9e7d-5d2c4510edd9","Type":"ContainerDied","Data":"388652c412c4a5a217c5dff9f305d6dee0ce2dc24d484c848184e0544916543a"} Oct 30 22:23:18.446543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.446481 5226 scope.go:122] "RemoveContainer" containerID="c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8" Oct 30 22:23:18.447940 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.447918 5226 generic.go:356] "Generic (PLEG): container finished" podID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerID="5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9" exitCode=0 Oct 30 22:23:18.448053 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.447958 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" Oct 30 22:23:18.448053 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.447982 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" event={"ID":"c1fdc873-78c7-404f-b22a-1b78df80c185","Type":"ContainerDied","Data":"5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9"} Oct 30 22:23:18.448053 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.448037 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj" event={"ID":"c1fdc873-78c7-404f-b22a-1b78df80c185","Type":"ContainerDied","Data":"955dd8b6f3f0abdb316acdeb0fb39c603ba823cd090a9d71b4bd3f5b36ae98b6"} Oct 30 22:23:18.456704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.456682 5226 scope.go:122] "RemoveContainer" containerID="c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8" Oct 30 22:23:18.457066 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:23:18.457038 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8\": container with ID starting with c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8 not found: ID does not exist" containerID="c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8" Oct 30 22:23:18.457152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.457073 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8"} err="failed to get container status \"c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8\": rpc error: code = NotFound desc = could not find container \"c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8\": container with ID starting with c624e9d01ac0b57438e52a42c196eefff7dc084e3af3f510b4adb40c5d533bb8 not found: ID does not exist" Oct 30 22:23:18.457152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.457092 5226 scope.go:122] "RemoveContainer" containerID="5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9" Oct 30 22:23:18.465939 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.465924 5226 scope.go:122] "RemoveContainer" containerID="5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9" Oct 30 22:23:18.466334 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:23:18.466309 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9\": container with ID starting with 5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9 not found: ID does not exist" containerID="5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9" Oct 30 22:23:18.466388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.466342 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9"} err="failed to get container status \"5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9\": rpc error: code = NotFound desc = could not find container \"5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9\": container with ID starting with 5e18313ce3400a4f9096ad1dacd4e29542234403673de24f8fa339fcfe6f3ad9 not found: ID does not exist" Oct 30 22:23:18.601569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.601532 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj"] Oct 30 22:23:18.647554 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.647529 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-g7jrj"] Oct 30 22:23:18.708232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.708155 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p"] Oct 30 22:23:18.714163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.714140 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-cv64p"] Oct 30 22:23:18.906698 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.906664 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" path="/var/lib/kubelet/pods/96c75851-785f-4799-9e7d-5d2c4510edd9/volumes" Oct 30 22:23:18.907153 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:18.907138 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" path="/var/lib/kubelet/pods/c1fdc873-78c7-404f-b22a-1b78df80c185/volumes" Oct 30 22:23:19.755746 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.755716 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw"] Oct 30 22:23:19.756350 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.756334 5226 cpu_manager.go:414] "containerMap: removing container" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" Oct 30 22:23:19.756400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.756351 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" Oct 30 22:23:19.756400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.756390 5226 cpu_manager.go:414] "containerMap: removing container" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" Oct 30 22:23:19.756400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.756395 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" Oct 30 22:23:19.756503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.756496 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c75851-785f-4799-9e7d-5d2c4510edd9" containerName="istio-proxy" Oct 30 22:23:19.756537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.756505 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fdc873-78c7-404f-b22a-1b78df80c185" containerName="istio-proxy" Oct 30 22:23:19.761523 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.761504 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.823376 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.823325 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6"] Oct 30 22:23:19.832674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.832606 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.861005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.860976 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw"] Oct 30 22:23:19.895266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.870067 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7ng2\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-kube-api-access-w7ng2\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.895619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.895588 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25edfa7-7bde-4b24-8ff8-306783a3f158-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.895878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.895859 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.896058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.896041 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.896175 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.896161 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.896474 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.896452 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.896666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.896649 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.896775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.896761 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.896930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.896899 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.931372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.928660 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6"] Oct 30 22:23:19.998618 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998583 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.998863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998631 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.998863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998707 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7ng2\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-kube-api-access-w7ng2\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.998863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998737 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.998863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998764 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25edfa7-7bde-4b24-8ff8-306783a3f158-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.999116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998874 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.999116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998913 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdw7\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-kube-api-access-9sdw7\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.999116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998955 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.999116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.998994 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.999116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999034 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.999116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999071 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.999116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999107 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf266f4f-e337-4e0e-a852-17467ece8400-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.999489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999154 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.999489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999264 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.999489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999306 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.999489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999342 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf266f4f-e337-4e0e-a852-17467ece8400-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:19.999489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999388 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:19.999489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:19.999411 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.000361 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.000314 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.001294 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.001228 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25edfa7-7bde-4b24-8ff8-306783a3f158-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.001482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.001463 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.003061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.003039 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.006888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.003939 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.006888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.006648 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.016716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.016694 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.060485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.060451 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7ng2\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-kube-api-access-w7ng2\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.065268 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.065240 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.078170 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.078140 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:20.097980 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.096113 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7"] Oct 30 22:23:20.101105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101058 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101118 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf266f4f-e337-4e0e-a852-17467ece8400-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101193 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101287 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101409 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdw7\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-kube-api-access-9sdw7\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101448 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101486 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101551 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf266f4f-e337-4e0e-a852-17467ece8400-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101594 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.101860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101677 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.101918 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101896 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-certs\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.102029 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.101953 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-data\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.103177 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.102432 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.103177 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.102594 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf266f4f-e337-4e0e-a852-17467ece8400-istiod-ca-cert\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.103177 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.102605 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-credential-socket\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.105623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.105603 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf266f4f-e337-4e0e-a852-17467ece8400-istio-podinfo\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.107620 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.107506 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-envoy\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.165228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.162776 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7"] Oct 30 22:23:20.190226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.178346 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-istio-token\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203452 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203549 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203586 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203660 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhd2\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-kube-api-access-2dhd2\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203692 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203783 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127b96df-63bc-4a6c-8bb8-f92de5603857-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203844 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203874 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.203938 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.213275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.204958 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdw7\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-kube-api-access-9sdw7\") pod \"e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.306886 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhd2\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-kube-api-access-2dhd2\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.306943 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.307035 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127b96df-63bc-4a6c-8bb8-f92de5603857-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.307103 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.307133 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.307196 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.307349 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.307419 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.307454 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.307902 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-data\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.308652 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-credential-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.309729 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127b96df-63bc-4a6c-8bb8-f92de5603857-istiod-ca-cert\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.309972 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-certs\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.314238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.310639 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-socket\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.319146 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.319094 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-podinfo\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.321926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.321877 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-envoy\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.396938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.393333 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-token\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.400807 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.400725 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhd2\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-kube-api-access-2dhd2\") pod \"e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.406075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.405481 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9"] Oct 30 22:23:20.416705 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.416673 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.440928 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.440898 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:20.460602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.460401 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:20.463737 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.463357 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9"] Oct 30 22:23:20.511788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.511753 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.511957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.511809 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.511957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.511883 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.511957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.511916 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.511957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.511955 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.512163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.511980 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsp6b\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-kube-api-access-bsp6b\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.512163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.512015 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.512163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.512064 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.512163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.512087 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.613988 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.613951 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614104 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614015 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614104 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614059 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614104 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614091 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsp6b\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-kube-api-access-bsp6b\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614143 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614348 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614397 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614427 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-data\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614553 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614643 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614622 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614786 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-certs\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614818 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istiod-ca-cert\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.614981 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.614962 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-credential-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.615042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.615021 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-socket\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.617758 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.617730 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-podinfo\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.618387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.618354 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-envoy\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.648399 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.648363 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw"] Oct 30 22:23:20.656707 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:23:20.656673 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25edfa7_7bde_4b24_8ff8_306783a3f158.slice/crio-1267f02d0028d824f00c3f67928933e0d7734da6613742001c55ed7906043807 WatchSource:0}: Error finding container 1267f02d0028d824f00c3f67928933e0d7734da6613742001c55ed7906043807: Status 404 returned error can't find the container with id 1267f02d0028d824f00c3f67928933e0d7734da6613742001c55ed7906043807 Oct 30 22:23:20.659267 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.659229 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:20.659358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.659301 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:20.659358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.659336 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:20.672371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.672348 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-token\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.672467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.672440 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsp6b\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-kube-api-access-bsp6b\") pod \"e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.739811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.739779 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:20.877374 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.877139 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7"] Oct 30 22:23:20.881863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.881834 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:20.881969 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.881943 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:20.882031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.882003 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:20.918761 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.918678 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6"] Oct 30 22:23:20.926683 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:23:20.926651 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf266f4f_e337_4e0e_a852_17467ece8400.slice/crio-5c4be099d6d5bded0700b7da1a74e40ce026f135009ea2dd3e86de247e0c7aa6 WatchSource:0}: Error finding container 5c4be099d6d5bded0700b7da1a74e40ce026f135009ea2dd3e86de247e0c7aa6: Status 404 returned error can't find the container with id 5c4be099d6d5bded0700b7da1a74e40ce026f135009ea2dd3e86de247e0c7aa6 Oct 30 22:23:20.931011 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.930977 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:20.931237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.931192 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:20.931382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:20.931367 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.198942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.198902 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9"] Oct 30 22:23:21.208077 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:23:21.207998 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a279b6_59b5_4ccc_8f77_8589e7eac9f2.slice/crio-c7938ed6347a6fc98487f8ace0cd9d9c60def24fbc4d905827676306ba12975b WatchSource:0}: Error finding container c7938ed6347a6fc98487f8ace0cd9d9c60def24fbc4d905827676306ba12975b: Status 404 returned error can't find the container with id c7938ed6347a6fc98487f8ace0cd9d9c60def24fbc4d905827676306ba12975b Oct 30 22:23:21.212063 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.212022 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.212331 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.212303 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.212531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.212502 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:21.475155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.475061 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" event={"ID":"127b96df-63bc-4a6c-8bb8-f92de5603857","Type":"ContainerStarted","Data":"404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495"} Oct 30 22:23:21.475352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.475155 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" event={"ID":"127b96df-63bc-4a6c-8bb8-f92de5603857","Type":"ContainerStarted","Data":"37004b194c757c47d41b3b19a27560082a71cda1c0d6cbc2fd8c9c05bcfed75a"} Oct 30 22:23:21.477354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.477321 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" event={"ID":"cf266f4f-e337-4e0e-a852-17467ece8400","Type":"ContainerStarted","Data":"084e2d25b42c0e5f11dff26e961e43d5f53ce0ad6d84fc9301b958173a4a802f"} Oct 30 22:23:21.477481 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.477377 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" event={"ID":"cf266f4f-e337-4e0e-a852-17467ece8400","Type":"ContainerStarted","Data":"5c4be099d6d5bded0700b7da1a74e40ce026f135009ea2dd3e86de247e0c7aa6"} Oct 30 22:23:21.485129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.484870 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" event={"ID":"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2","Type":"ContainerStarted","Data":"617c696bb9618977a9d588705f825780749a77f3c67569d2af91dd6d8aa75073"} Oct 30 22:23:21.485129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.484954 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" event={"ID":"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2","Type":"ContainerStarted","Data":"c7938ed6347a6fc98487f8ace0cd9d9c60def24fbc4d905827676306ba12975b"} Oct 30 22:23:21.486988 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.486938 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" event={"ID":"c25edfa7-7bde-4b24-8ff8-306783a3f158","Type":"ContainerStarted","Data":"67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39"} Oct 30 22:23:21.487115 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.487008 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" event={"ID":"c25edfa7-7bde-4b24-8ff8-306783a3f158","Type":"ContainerStarted","Data":"1267f02d0028d824f00c3f67928933e0d7734da6613742001c55ed7906043807"} Oct 30 22:23:21.589622 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.589571 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podStartSLOduration=1.5895576120000001 podStartE2EDuration="1.589557612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:21.588137993 +0000 UTC m=+1292.845479682" watchObservedRunningTime="2026-10-30 22:23:21.589557612 +0000 UTC m=+1292.846899322" Oct 30 22:23:21.713069 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.713006 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podStartSLOduration=1.712986967 podStartE2EDuration="1.712986967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:21.710027529 +0000 UTC m=+1292.967369241" watchObservedRunningTime="2026-10-30 22:23:21.712986967 +0000 UTC m=+1292.970328679" Oct 30 22:23:21.774951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.774874 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:21.971296 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:21.971196 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podStartSLOduration=2.971176058 podStartE2EDuration="2.971176058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:21.851231578 +0000 UTC m=+1293.108573285" watchObservedRunningTime="2026-10-30 22:23:21.971176058 +0000 UTC m=+1293.228517770" Oct 30 22:23:22.080289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.078600 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:22.088231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.085334 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:22.088231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.085413 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:22.459228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.452021 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:22.459228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.452155 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:22.459228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.452197 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:22.477229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.467401 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:22.477839 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.477807 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:22.478060 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.478032 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:22.746980 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.744110 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:22.746980 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:22.744180 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:23.094354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.083869 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:23.094354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.083928 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:23.575290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.453814 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:23.575290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.453895 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:23.575290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.463629 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:23.575290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.463684 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:23.832345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.817285 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:23:23.889257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.888355 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:23.889257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.888824 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 22:23:23.889516 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.889411 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:23.897236 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.893565 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:23.897236 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.893830 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 22:23:23.937230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.933521 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:23.937230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.933593 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:23.958228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:23.954227 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:24.127578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.127488 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:24.127826 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.127798 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:24.239231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.238380 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:24.443230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.442740 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:24.443230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.442832 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:24.497230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.485118 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:24.497230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.485222 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:24.617230 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:23:24.614707 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:23:24.745291 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.741748 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:24.745291 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.741839 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:24.827230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.826359 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:24.892230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.891351 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:24.892445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.892351 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:23:24.894232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.892698 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 22:23:24.894232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.893009 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 22:23:24.894232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.893301 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:24.898233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.894839 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:23:24.898233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.894917 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:23:24.898233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.894972 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:23:24.898233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.895007 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:23:24.919230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.914882 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 22:23:24.919230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.917374 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 30 22:23:24.942234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.929858 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 22:23:24.959240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.958402 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:25.000290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:24.987783 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:25.016229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.015174 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:23:25.016229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.015976 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:23:25.016229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.016027 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:23:25.061230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.060414 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:25.122230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.098288 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podStartSLOduration=6.098266267 podStartE2EDuration="6.098266267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:21.974785581 +0000 UTC m=+1293.232127292" watchObservedRunningTime="2026-10-30 22:23:25.098266267 +0000 UTC m=+1296.355607997" Oct 30 22:23:25.224229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.223537 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:25.224229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.223601 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:25.250412 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.249638 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:25.250412 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.250109 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:23:25.426417 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.426361 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:25.494434 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.494400 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:25.522549 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.494856 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:25.755230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.721340 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:25.755230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.721413 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:25.785599 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.785060 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:25.785599 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.785118 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:25.851628 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.843330 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:25.866771 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.866649 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:23:25.879261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.875978 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Oct 30 22:23:25.900821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.895036 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 22:23:25.902646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.895088 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:23:25.924654 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.895115 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 22:23:25.926058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.895135 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 22:23:25.960228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.895182 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 22:23:26.052218 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.920813 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:26.082229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.958298 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:23:26.082229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:25.961461 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 22:23:26.082229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.006344 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:26.082229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.009285 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 22:23:26.082229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.061377 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:23:26.082229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.062177 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:26.082229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.063858 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:23:26.184574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.176387 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.184574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.176445 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:26.256253 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.252310 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:26.302003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.301901 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 30 22:23:26.460254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.453363 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.460254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.453422 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:26.467249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.464577 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:26.467249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.465573 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.467249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.465614 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:26.757287 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.753374 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:26.973736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.757803 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:26.973736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.860709 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:26.973736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.898760 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:26.973736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.906613 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 22:23:26.973736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.926562 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:26.973736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.930444 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 22:23:26.987601 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:26.984307 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.093442 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.093333 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.205293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.110738 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 22:23:27.205293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.110782 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.205293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.110807 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.205293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.111178 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:27.205293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.164493 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:27.205293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.135974 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.205293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.135936 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.245476 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.233598 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:23:27.395297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.270361 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.412232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.340341 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.537497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.517432 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:27.537497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.517500 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:27.537497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.517857 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:27.537497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.517897 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:27.759750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.759586 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:27.759750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.759647 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:27.904364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.904326 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.916064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.910054 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.942136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.941309 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:27.942949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.942819 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Oct 30 22:23:27.943092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:27.943055 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.057094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.055330 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.113230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.109979 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:28.113230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.110033 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:28.173230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.170291 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 22:23:28.173546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.173509 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.173994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.173970 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 22:23:28.177520 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.175659 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.181320 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.175687 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 22:23:28.181746 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.175707 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 22:23:28.421222 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.421102 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:28.439433 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.439323 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.460981 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.460942 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:28.461383 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.461272 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:28.461920 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.461886 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:28.462079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.462059 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:28.571014 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.570981 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:23:28.740634 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.740598 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:28.740919 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.740889 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:28.897258 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.897226 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:23:28.927617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.927547 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.950496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.950439 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:28.976141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:28.976098 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 22:23:29.059653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.059543 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.112364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.087079 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:29.112364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.087155 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:29.196136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.196092 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 22:23:29.197456 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.197417 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 22:23:29.218227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.214312 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.237710 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.237650 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 22:23:29.239126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.239081 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 22:23:29.347479 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.347363 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Oct 30 22:23:29.435865 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.435816 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.608797 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.608710 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:29.633392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.633322 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:29.633799 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.608710 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:29.633966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.633938 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:29.660471 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.660428 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz"] Oct 30 22:23:29.768667 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.768628 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:29.774807 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.774763 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:29.775265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.775240 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz"] Oct 30 22:23:29.775480 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.775466 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:23:29.778060 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.776498 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:29.839125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.839090 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-05e3625f\"" Oct 30 22:23:29.840266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.840243 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-05e3625f-dockercfg-877c6\"" Oct 30 22:23:29.881289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.880416 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9dr\" (UniqueName: \"kubernetes.io/projected/255db097-fbbc-4d21-823b-d4dd796c30df-kube-api-access-mb9dr\") pod \"payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz\" (UID: \"255db097-fbbc-4d21-823b-d4dd796c30df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:29.881289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.880747 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255db097-fbbc-4d21-823b-d4dd796c30df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz\" (UID: \"255db097-fbbc-4d21-823b-d4dd796c30df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:29.929386 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.929337 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.977934 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.977894 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:29.979513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:29.979483 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 22:23:30.035229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.035166 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255db097-fbbc-4d21-823b-d4dd796c30df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz\" (UID: \"255db097-fbbc-4d21-823b-d4dd796c30df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:30.035438 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.035354 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9dr\" (UniqueName: \"kubernetes.io/projected/255db097-fbbc-4d21-823b-d4dd796c30df-kube-api-access-mb9dr\") pod \"payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz\" (UID: \"255db097-fbbc-4d21-823b-d4dd796c30df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:30.036495 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.036441 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255db097-fbbc-4d21-823b-d4dd796c30df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz\" (UID: \"255db097-fbbc-4d21-823b-d4dd796c30df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:30.089228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.064037 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 22:23:30.116282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.091996 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.116622 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.116584 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:30.117900 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.117878 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:23:30.138990 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.138890 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh"] Oct 30 22:23:30.220144 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.218315 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.220144 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.219461 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.222850 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.222659 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.248249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.248215 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.248567 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.248150 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:30.292027 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.291998 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:30.375250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.354598 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmdjh\" (UniqueName: \"kubernetes.io/projected/d0aa45f6-f151-4d93-b26e-99e6423c05cc-kube-api-access-pmdjh\") pod \"payload-processing-e2e-ait-05e3625f-5c977df99-tpplh\" (UID: \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\") " pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:30.375250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.354709 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aa45f6-f151-4d93-b26e-99e6423c05cc-plugins-config-volume\") pod \"payload-processing-e2e-ait-05e3625f-5c977df99-tpplh\" (UID: \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\") " pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:30.478409 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.478363 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh"] Oct 30 22:23:30.480160 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.480120 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmdjh\" (UniqueName: \"kubernetes.io/projected/d0aa45f6-f151-4d93-b26e-99e6423c05cc-kube-api-access-pmdjh\") pod \"payload-processing-e2e-ait-05e3625f-5c977df99-tpplh\" (UID: \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\") " pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:30.480479 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.480453 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aa45f6-f151-4d93-b26e-99e6423c05cc-plugins-config-volume\") pod \"payload-processing-e2e-ait-05e3625f-5c977df99-tpplh\" (UID: \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\") " pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:30.482113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.482051 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aa45f6-f151-4d93-b26e-99e6423c05cc-plugins-config-volume\") pod \"payload-processing-e2e-ait-05e3625f-5c977df99-tpplh\" (UID: \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\") " pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:30.486714 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.486595 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.486714 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.486651 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:30.504231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.500842 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.512341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.512294 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:30.618247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.599378 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:30.683245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.669057 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmdjh\" (UniqueName: \"kubernetes.io/projected/d0aa45f6-f151-4d93-b26e-99e6423c05cc-kube-api-access-pmdjh\") pod \"payload-processing-e2e-ait-05e3625f-5c977df99-tpplh\" (UID: \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\") " pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:30.760293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.741651 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:30.760293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.741715 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:30.891249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.869845 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:30.933258 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:30.930781 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.176442 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.170197 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.176442 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.170568 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 22:23:31.202695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.202658 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 22:23:31.202985 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.202951 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 22:23:31.240230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.232672 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.262241 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.258755 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.279243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.259183 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 22:23:31.293829 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.259240 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 22:23:31.294432 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.259265 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 22:23:31.294916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.259309 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.307881 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.259330 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:31.396250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.394883 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:23:31.396250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.394955 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:23:31.444763 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.438923 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9dr\" (UniqueName: \"kubernetes.io/projected/255db097-fbbc-4d21-823b-d4dd796c30df-kube-api-access-mb9dr\") pod \"payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz\" (UID: \"255db097-fbbc-4d21-823b-d4dd796c30df\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:31.444763 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.441599 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:31.444763 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.441658 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:31.469759 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.469720 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:23:31.470066 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.470034 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 22:23:31.486890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.486858 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:23:31.576859 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.574499 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:23:31.599666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.599592 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:31.666329 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.661807 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:32.009231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.844361 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:32.009231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.844438 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:32.009231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:31.953196 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:23:32.210618 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.210577 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 22:23:32.211154 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.211129 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 22:23:32.552290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.529126 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 22:23:32.611717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.586846 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:23:32.612157 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.587265 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 30 22:23:32.612484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.598394 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 22:23:32.612724 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.612698 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 22:23:32.612958 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.598961 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.613643 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.598983 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.614095 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.599006 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 22:23:32.614536 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.599707 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 22:23:32.614951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.600497 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.618570 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.598926 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.618687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.618619 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:23:32.623507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.623384 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:32.623507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.623435 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:32.790122 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.781106 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:32.790122 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.781159 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:32.841354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:32.833293 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:23:33.149290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.125952 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.149290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.126365 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 22:23:33.251143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.217591 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 22:23:33.354240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.316896 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.354240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.336662 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:33.354240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.336750 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.648234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.639311 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 22:23:33.648234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.640911 5226 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.695228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.688370 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:33.695228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.690694 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:33.695228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:33.690752 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.007249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.006244 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 22:23:34.007249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.006297 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 22:23:34.074437 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.054949 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 22:23:34.074437 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.074392 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 22:23:34.076436 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.075736 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 22:23:34.099270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.075788 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 22:23:34.137261 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:23:34.119669 5226 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.192s" Oct 30 22:23:34.137607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.137584 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:23:34.137864 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.137853 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:23:34.218232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.218157 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 22:23:34.219753 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.219730 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:23:34.313688 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.313644 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 22:23:34.314018 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.313986 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded" Oct 30 22:23:34.314247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.314221 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.399086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.399040 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 22:23:34.450680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.450650 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:23:34.506942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.505910 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:34.506942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.505982 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.517632 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.517599 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:34.518867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.518846 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:34.529632 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.529607 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:23:34.547778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.547360 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:34.547778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.547433 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.551727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.551695 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" event={"ID":"255db097-fbbc-4d21-823b-d4dd796c30df","Type":"ContainerStarted","Data":"fff892223318638062af20e700dbbe98ba50a5bc679fa086ecbcd1acc4a80ed8"} Oct 30 22:23:34.677660 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.673391 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.709237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.702861 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 22:23:34.709846 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.709656 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" event={"ID":"d0aa45f6-f151-4d93-b26e-99e6423c05cc","Type":"ContainerStarted","Data":"d2cf2247c4ddb9c504bdd4b6397eab22e85ef4fbe586e3a0163ba011f971f35f"} Oct 30 22:23:34.710168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.710138 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:34.710347 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.710327 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:34.781276 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:34.759430 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.089893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.083116 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.089893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.087099 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 22:23:35.089893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.089660 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.126576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.091421 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.211292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.211259 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:23:35.211571 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.211558 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:23:35.211668 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.211658 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh"] Oct 30 22:23:35.211794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.211785 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:23:35.211880 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.211872 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz"] Oct 30 22:23:35.211992 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.211983 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:23:35.212094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.212086 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:23:35.246860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.246788 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:23:35.248686 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.248662 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:23:35.249024 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.248875 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.249606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.249591 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:23:35.255783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.255711 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:23:35.308612 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.308585 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:23:35.373912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.373790 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:23:35.446568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.446480 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:35.463813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.463785 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:35.549263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.549092 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:35.549263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.549157 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.718476 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.718428 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" event={"ID":"d0aa45f6-f151-4d93-b26e-99e6423c05cc","Type":"ContainerStarted","Data":"1cd90441d7f45ea69c61be241ce701d9e262bb3a3e8b9df7b23cc18dca59e111"} Oct 30 22:23:35.719166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.719142 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:35.721086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.721055 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" event={"ID":"255db097-fbbc-4d21-823b-d4dd796c30df","Type":"ContainerStarted","Data":"19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300"} Oct 30 22:23:35.721292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.721274 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:35.771225 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.747326 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:35.810226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.783301 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:35.810226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.783368 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:35.812255 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.812224 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:35.812538 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:35.812506 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.530681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.530651 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm"] Oct 30 22:23:36.541154 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.540254 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:36.637227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.590715 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ae20a62d\"" Oct 30 22:23:36.637227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.590875 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ae20a62d-dockercfg-bftl9\"" Oct 30 22:23:36.645246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.637570 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:36.645246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.637634 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.645246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.638980 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072daa7-79bd-4374-90fe-50b99315006d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm\" (UID: \"b072daa7-79bd-4374-90fe-50b99315006d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:36.645246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.639133 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vljw2\" (UniqueName: \"kubernetes.io/projected/b072daa7-79bd-4374-90fe-50b99315006d-kube-api-access-vljw2\") pod \"payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm\" (UID: \"b072daa7-79bd-4374-90fe-50b99315006d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:36.678235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.677737 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:36.680800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.680776 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:23:36.767990 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.767826 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072daa7-79bd-4374-90fe-50b99315006d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm\" (UID: \"b072daa7-79bd-4374-90fe-50b99315006d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:36.768590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.768564 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljw2\" (UniqueName: \"kubernetes.io/projected/b072daa7-79bd-4374-90fe-50b99315006d-kube-api-access-vljw2\") pod \"payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm\" (UID: \"b072daa7-79bd-4374-90fe-50b99315006d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:36.771296 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.770112 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072daa7-79bd-4374-90fe-50b99315006d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm\" (UID: \"b072daa7-79bd-4374-90fe-50b99315006d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:36.824820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.824318 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm"] Oct 30 22:23:36.841716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.841407 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs"] Oct 30 22:23:36.852839 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.852808 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:36.858105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.854482 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:23:36.858105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.854554 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:23:36.901075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.901044 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljw2\" (UniqueName: \"kubernetes.io/projected/b072daa7-79bd-4374-90fe-50b99315006d-kube-api-access-vljw2\") pod \"payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm\" (UID: \"b072daa7-79bd-4374-90fe-50b99315006d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:36.917545 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:36.917514 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs"] Oct 30 22:23:37.000699 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.000651 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10a43fd-f5a8-4488-8c02-c9adb876a639-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs\" (UID: \"d10a43fd-f5a8-4488-8c02-c9adb876a639\") " pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:37.002014 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.001988 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2gk4\" (UniqueName: \"kubernetes.io/projected/d10a43fd-f5a8-4488-8c02-c9adb876a639-kube-api-access-c2gk4\") pod \"payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs\" (UID: \"d10a43fd-f5a8-4488-8c02-c9adb876a639\") " pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:37.113329 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.105324 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2gk4\" (UniqueName: \"kubernetes.io/projected/d10a43fd-f5a8-4488-8c02-c9adb876a639-kube-api-access-c2gk4\") pod \"payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs\" (UID: \"d10a43fd-f5a8-4488-8c02-c9adb876a639\") " pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:37.113329 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.105659 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10a43fd-f5a8-4488-8c02-c9adb876a639-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs\" (UID: \"d10a43fd-f5a8-4488-8c02-c9adb876a639\") " pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:37.113329 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.106704 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10a43fd-f5a8-4488-8c02-c9adb876a639-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs\" (UID: \"d10a43fd-f5a8-4488-8c02-c9adb876a639\") " pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:37.196852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.196776 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" podStartSLOduration=8.196754903 podStartE2EDuration="8.196754903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:37.067743822 +0000 UTC m=+1308.325085534" watchObservedRunningTime="2026-10-30 22:23:37.196754903 +0000 UTC m=+1308.454096614" Oct 30 22:23:37.200450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.200178 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:37.208343 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.208306 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2gk4\" (UniqueName: \"kubernetes.io/projected/d10a43fd-f5a8-4488-8c02-c9adb876a639-kube-api-access-c2gk4\") pod \"payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs\" (UID: \"d10a43fd-f5a8-4488-8c02-c9adb876a639\") " pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:37.227274 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.227245 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:37.285136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.285086 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:23:37.299463 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.298943 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:37.363695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.363572 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" podStartSLOduration=8.36355091 podStartE2EDuration="8.36355091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:37.355328661 +0000 UTC m=+1308.612670384" watchObservedRunningTime="2026-10-30 22:23:37.36355091 +0000 UTC m=+1308.620892620" Oct 30 22:23:37.517133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.515586 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:37.521261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.520362 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:23:37.731381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.731349 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm"] Oct 30 22:23:37.738425 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:23:37.738380 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb072daa7_79bd_4374_90fe_50b99315006d.slice/crio-824c132727b6ae5437e0495176c3a201c0a792f4138d0f580167032903aeea15 WatchSource:0}: Error finding container 824c132727b6ae5437e0495176c3a201c0a792f4138d0f580167032903aeea15: Status 404 returned error can't find the container with id 824c132727b6ae5437e0495176c3a201c0a792f4138d0f580167032903aeea15 Oct 30 22:23:37.772897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.772866 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs"] Oct 30 22:23:37.779980 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:23:37.779949 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10a43fd_f5a8_4488_8c02_c9adb876a639.slice/crio-ed36ecebe83c4a920c91e2667c3763ce438360f1178a66e8d383d550b1ba5534 WatchSource:0}: Error finding container ed36ecebe83c4a920c91e2667c3763ce438360f1178a66e8d383d550b1ba5534: Status 404 returned error can't find the container with id ed36ecebe83c4a920c91e2667c3763ce438360f1178a66e8d383d550b1ba5534 Oct 30 22:23:37.793481 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.793450 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" event={"ID":"d10a43fd-f5a8-4488-8c02-c9adb876a639","Type":"ContainerStarted","Data":"ed36ecebe83c4a920c91e2667c3763ce438360f1178a66e8d383d550b1ba5534"} Oct 30 22:23:37.795964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:37.795919 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" event={"ID":"b072daa7-79bd-4374-90fe-50b99315006d","Type":"ContainerStarted","Data":"824c132727b6ae5437e0495176c3a201c0a792f4138d0f580167032903aeea15"} Oct 30 22:23:38.804527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:38.804493 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" event={"ID":"b072daa7-79bd-4374-90fe-50b99315006d","Type":"ContainerStarted","Data":"226cf36a8182aad47c6f4838f8cdb41ec2ecf8fe7b6d0521e4d5d0ad8a631529"} Oct 30 22:23:38.805053 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:38.804593 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:38.806261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:38.806232 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" event={"ID":"d10a43fd-f5a8-4488-8c02-c9adb876a639","Type":"ContainerStarted","Data":"da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed"} Oct 30 22:23:38.806419 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:38.806369 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:38.917803 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:38.917755 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" podStartSLOduration=2.917741483 podStartE2EDuration="2.917741483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:38.917171932 +0000 UTC m=+1310.174513641" watchObservedRunningTime="2026-10-30 22:23:38.917741483 +0000 UTC m=+1310.175083192" Oct 30 22:23:39.028960 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:39.028898 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" podStartSLOduration=3.028885701 podStartE2EDuration="3.028885701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:39.028127601 +0000 UTC m=+1310.285469311" watchObservedRunningTime="2026-10-30 22:23:39.028885701 +0000 UTC m=+1310.286227410" Oct 30 22:23:44.102063 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.102027 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv"] Oct 30 22:23:44.106576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.106551 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:44.157007 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.156472 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dee83803\"" Oct 30 22:23:44.201666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.201630 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dee83803-dockercfg-5g5sk\"" Oct 30 22:23:44.207552 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.207522 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv"] Oct 30 22:23:44.228598 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.228561 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f65b40-ed93-4cbf-8dc9-03327d473926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv\" (UID: \"b4f65b40-ed93-4cbf-8dc9-03327d473926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:44.228791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.228619 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62svv\" (UniqueName: \"kubernetes.io/projected/b4f65b40-ed93-4cbf-8dc9-03327d473926-kube-api-access-62svv\") pod \"payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv\" (UID: \"b4f65b40-ed93-4cbf-8dc9-03327d473926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:44.272418 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.266071 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg"] Oct 30 22:23:44.272418 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.271422 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:44.332292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.330873 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f65b40-ed93-4cbf-8dc9-03327d473926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv\" (UID: \"b4f65b40-ed93-4cbf-8dc9-03327d473926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:44.332292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.330924 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62svv\" (UniqueName: \"kubernetes.io/projected/b4f65b40-ed93-4cbf-8dc9-03327d473926-kube-api-access-62svv\") pod \"payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv\" (UID: \"b4f65b40-ed93-4cbf-8dc9-03327d473926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:44.332292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.331118 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d5470-9294-41dd-8998-0663648e4197-plugins-config-volume\") pod \"payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg\" (UID: \"1f3d5470-9294-41dd-8998-0663648e4197\") " pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:44.332292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.331244 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89pjk\" (UniqueName: \"kubernetes.io/projected/1f3d5470-9294-41dd-8998-0663648e4197-kube-api-access-89pjk\") pod \"payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg\" (UID: \"1f3d5470-9294-41dd-8998-0663648e4197\") " pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:44.349238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.346100 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg"] Oct 30 22:23:44.349238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.348431 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f65b40-ed93-4cbf-8dc9-03327d473926-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv\" (UID: \"b4f65b40-ed93-4cbf-8dc9-03327d473926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:44.418255 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.418112 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62svv\" (UniqueName: \"kubernetes.io/projected/b4f65b40-ed93-4cbf-8dc9-03327d473926-kube-api-access-62svv\") pod \"payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv\" (UID: \"b4f65b40-ed93-4cbf-8dc9-03327d473926\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:44.423376 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.423341 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:44.432576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.432543 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d5470-9294-41dd-8998-0663648e4197-plugins-config-volume\") pod \"payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg\" (UID: \"1f3d5470-9294-41dd-8998-0663648e4197\") " pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:44.432726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.432615 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89pjk\" (UniqueName: \"kubernetes.io/projected/1f3d5470-9294-41dd-8998-0663648e4197-kube-api-access-89pjk\") pod \"payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg\" (UID: \"1f3d5470-9294-41dd-8998-0663648e4197\") " pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:44.433351 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.433327 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d5470-9294-41dd-8998-0663648e4197-plugins-config-volume\") pod \"payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg\" (UID: \"1f3d5470-9294-41dd-8998-0663648e4197\") " pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:44.491363 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.491304 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89pjk\" (UniqueName: \"kubernetes.io/projected/1f3d5470-9294-41dd-8998-0663648e4197-kube-api-access-89pjk\") pod \"payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg\" (UID: \"1f3d5470-9294-41dd-8998-0663648e4197\") " pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:44.591052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.588262 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:44.813764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.813731 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv"] Oct 30 22:23:44.819252 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:23:44.819197 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f65b40_ed93_4cbf_8dc9_03327d473926.slice/crio-6b7ed81eedba8e6322d2489f4a1bc3cb4443b4d578439b7e6bbd389e59ef6fcd WatchSource:0}: Error finding container 6b7ed81eedba8e6322d2489f4a1bc3cb4443b4d578439b7e6bbd389e59ef6fcd: Status 404 returned error can't find the container with id 6b7ed81eedba8e6322d2489f4a1bc3cb4443b4d578439b7e6bbd389e59ef6fcd Oct 30 22:23:44.876680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.876654 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" event={"ID":"b4f65b40-ed93-4cbf-8dc9-03327d473926","Type":"ContainerStarted","Data":"6b7ed81eedba8e6322d2489f4a1bc3cb4443b4d578439b7e6bbd389e59ef6fcd"} Oct 30 22:23:44.981986 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:44.981950 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg"] Oct 30 22:23:44.987389 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:23:44.987357 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3d5470_9294_41dd_8998_0663648e4197.slice/crio-be689265059b9994722df0afc3a705a2ec29946781e05d87c330f9106af60e12 WatchSource:0}: Error finding container be689265059b9994722df0afc3a705a2ec29946781e05d87c330f9106af60e12: Status 404 returned error can't find the container with id be689265059b9994722df0afc3a705a2ec29946781e05d87c330f9106af60e12 Oct 30 22:23:45.885288 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:45.885250 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" event={"ID":"b4f65b40-ed93-4cbf-8dc9-03327d473926","Type":"ContainerStarted","Data":"131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3"} Oct 30 22:23:45.885687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:45.885332 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:45.887022 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:45.886995 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" event={"ID":"1f3d5470-9294-41dd-8998-0663648e4197","Type":"ContainerStarted","Data":"86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c"} Oct 30 22:23:45.887133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:45.887044 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" event={"ID":"1f3d5470-9294-41dd-8998-0663648e4197","Type":"ContainerStarted","Data":"be689265059b9994722df0afc3a705a2ec29946781e05d87c330f9106af60e12"} Oct 30 22:23:45.887133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:45.887065 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:45.997832 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:45.997776 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" podStartSLOduration=1.997764227 podStartE2EDuration="1.997764227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:45.994641756 +0000 UTC m=+1317.251983466" watchObservedRunningTime="2026-10-30 22:23:45.997764227 +0000 UTC m=+1317.255105936" Oct 30 22:23:46.103455 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:46.103398 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" podStartSLOduration=2.103383309 podStartE2EDuration="2.103383309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:46.103345752 +0000 UTC m=+1317.360687460" watchObservedRunningTime="2026-10-30 22:23:46.103383309 +0000 UTC m=+1317.360725019" Oct 30 22:23:46.820172 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:46.820143 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:23:47.799166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:47.799134 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:23:49.816542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:49.816512 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:23:49.816995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:49.816593 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:23:51.490142 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.490105 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj"] Oct 30 22:23:51.496100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.496073 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.549519 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.549489 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjsx\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-kube-api-access-vpjsx\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.549721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.549599 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.549721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.549685 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.549799 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.549730 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.549834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.549824 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.549868 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.549849 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.549991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.549969 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.550029 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.550014 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.550114 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.550101 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.556560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.556534 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj"] Oct 30 22:23:51.651091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651057 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651117 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651172 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651256 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651284 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651389 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651440 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651513 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651540 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-data\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.651616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651555 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjsx\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-kube-api-access-vpjsx\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.652250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.651988 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.652250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.652030 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-credential-socket\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.652250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.652185 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-certs\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.652438 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.652304 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istiod-ca-cert\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.654060 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.654033 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-podinfo\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.654188 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.654082 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-envoy\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.709654 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.709618 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-token\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.709798 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.709775 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjsx\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-kube-api-access-vpjsx\") pod \"e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:51.811661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:51.811576 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:52.242444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:52.242411 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj"] Oct 30 22:23:52.248596 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:23:52.248549 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90044cb5_a7fd_47ac_8db9_ed74f0b7cd72.slice/crio-2f9a36813f3a1ab4b82751d4a03f19b1f87da92fe03f30c1935da9fa4ad6a5e4 WatchSource:0}: Error finding container 2f9a36813f3a1ab4b82751d4a03f19b1f87da92fe03f30c1935da9fa4ad6a5e4: Status 404 returned error can't find the container with id 2f9a36813f3a1ab4b82751d4a03f19b1f87da92fe03f30c1935da9fa4ad6a5e4 Oct 30 22:23:52.254398 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:52.254361 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:52.254522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:52.254444 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:52.254522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:52.254491 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:23:52.956567 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:52.956523 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" event={"ID":"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72","Type":"ContainerStarted","Data":"a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e"} Oct 30 22:23:52.957043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:52.956590 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" event={"ID":"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72","Type":"ContainerStarted","Data":"2f9a36813f3a1ab4b82751d4a03f19b1f87da92fe03f30c1935da9fa4ad6a5e4"} Oct 30 22:23:53.162520 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:53.162465 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podStartSLOduration=2.162450213 podStartE2EDuration="2.162450213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:53.115883562 +0000 UTC m=+1324.373225271" watchObservedRunningTime="2026-10-30 22:23:53.162450213 +0000 UTC m=+1324.419791923" Oct 30 22:23:53.812522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:53.812487 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:53.814070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:53.814044 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:23:53.814198 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:53.814089 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:23:54.414966 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:23:54.414929 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:23:54.812376 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:54.812342 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:23:54.812568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:54.812397 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:23:55.812006 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:55.811973 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:23:55.812461 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:55.812026 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:23:56.812698 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:56.812655 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:23:56.813092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:56.812708 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:23:56.895943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:56.895915 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:23:56.896114 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:56.895987 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:23:57.812403 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:57.812367 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:23:57.812565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:57.812430 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:23:58.812619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:58.812576 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:23:58.813116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:58.812636 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 22:23:59.816871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:59.816842 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:59.817315 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:59.817254 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:23:59.817943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:23:59.817923 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:24:24.414068 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:24:24.414029 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:24:28.198628 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.198594 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg"] Oct 30 22:24:28.204813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.204789 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.244968 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.244934 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd"] Oct 30 22:24:28.249769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.249746 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.269292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.260499 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8ctw4\"" Oct 30 22:24:28.311645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.311613 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg"] Oct 30 22:24:28.317810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.317771 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd"] Oct 30 22:24:28.366933 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.366814 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.366933 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.366862 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a49049-a92f-4eeb-aa52-ff15da31e664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.366933 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.366890 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.367314 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367031 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.367314 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367075 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44z4q\" (UniqueName: \"kubernetes.io/projected/82a49049-a92f-4eeb-aa52-ff15da31e664-kube-api-access-44z4q\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.367314 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367129 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.367314 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367177 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d68e5b-715f-4393-a12a-f06477f82928-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.367314 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367293 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqlm\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-kube-api-access-mnqlm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.367590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367333 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.367590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367433 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.367590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367472 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.367590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367560 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d68e5b-715f-4393-a12a-f06477f82928-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.367797 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367690 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.367857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367799 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.367912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367863 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.367971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.367955 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.368055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.368033 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.368127 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.368064 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469004 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469086 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469111 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469169 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469237 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a49049-a92f-4eeb-aa52-ff15da31e664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469265 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469386 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469440 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44z4q\" (UniqueName: \"kubernetes.io/projected/82a49049-a92f-4eeb-aa52-ff15da31e664-kube-api-access-44z4q\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469477 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469503 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d68e5b-715f-4393-a12a-f06477f82928-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469532 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469543 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqlm\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-kube-api-access-mnqlm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469580 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469614 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469621 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469670 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.469959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469704 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469713 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469767 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d68e5b-715f-4393-a12a-f06477f82928-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.469959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469906 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.470392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.469989 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.470392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.470013 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.470392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.470044 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.470392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.470076 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.470392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.470042 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a49049-a92f-4eeb-aa52-ff15da31e664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.470392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.470369 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.470781 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.470530 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.470781 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.470674 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d68e5b-715f-4393-a12a-f06477f82928-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.472177 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.472151 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.472312 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.472251 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.472371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.472361 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d68e5b-715f-4393-a12a-f06477f82928-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.472422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.472362 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.527183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.527147 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.527612 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.527586 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqlm\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-kube-api-access-mnqlm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-m55sg\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:28.527704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.527638 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a49049-a92f-4eeb-aa52-ff15da31e664-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.532024 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.531995 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44z4q\" (UniqueName: \"kubernetes.io/projected/82a49049-a92f-4eeb-aa52-ff15da31e664-kube-api-access-44z4q\") pod \"maas-default-gateway-openshift-default-75bf867b6c-hfhbd\" (UID: \"82a49049-a92f-4eeb-aa52-ff15da31e664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.585822 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.585785 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:28.820859 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:28.820779 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:29.018217 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.018181 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd"] Oct 30 22:24:29.023389 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:24:29.023358 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a49049_a92f_4eeb_aa52_ff15da31e664.slice/crio-307e38bdbeb54a92d83526a7cf1b3e89c12ce7a29e3571ef6972f7eb5c639bdc WatchSource:0}: Error finding container 307e38bdbeb54a92d83526a7cf1b3e89c12ce7a29e3571ef6972f7eb5c639bdc: Status 404 returned error can't find the container with id 307e38bdbeb54a92d83526a7cf1b3e89c12ce7a29e3571ef6972f7eb5c639bdc Oct 30 22:24:29.025915 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.025881 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:29.026005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.025948 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:29.026005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.025974 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:29.207529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.207497 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg"] Oct 30 22:24:29.212427 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:24:29.212399 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d68e5b_715f_4393_a12a_f06477f82928.slice/crio-ff8f960470de3e17d2024b7e005f3684d47372801be2297c4faddcf4bde6a0b8 WatchSource:0}: Error finding container ff8f960470de3e17d2024b7e005f3684d47372801be2297c4faddcf4bde6a0b8: Status 404 returned error can't find the container with id ff8f960470de3e17d2024b7e005f3684d47372801be2297c4faddcf4bde6a0b8 Oct 30 22:24:29.215712 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.215674 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:29.215813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.215758 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:29.215813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.215799 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:29.216848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.216825 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" event={"ID":"82a49049-a92f-4eeb-aa52-ff15da31e664","Type":"ContainerStarted","Data":"214905c2eda0d45c903dc0dc0eda334dcd1639ddb26e9c3ee03d6bfe70051233"} Oct 30 22:24:29.216937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.216880 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" event={"ID":"82a49049-a92f-4eeb-aa52-ff15da31e664","Type":"ContainerStarted","Data":"307e38bdbeb54a92d83526a7cf1b3e89c12ce7a29e3571ef6972f7eb5c639bdc"} Oct 30 22:24:29.336018 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.335960 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podStartSLOduration=1.33594393 podStartE2EDuration="1.33594393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:29.332819527 +0000 UTC m=+1360.590161239" watchObservedRunningTime="2026-10-30 22:24:29.33594393 +0000 UTC m=+1360.593285639" Oct 30 22:24:29.586892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:29.586860 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:30.270229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:30.270072 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" event={"ID":"81d68e5b-715f-4393-a12a-f06477f82928","Type":"ContainerStarted","Data":"a04ebb720c83e37e507027d5e4a26419b6baf5aff3243e4d3986c0e42d11f272"} Oct 30 22:24:30.270229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:30.270134 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" event={"ID":"81d68e5b-715f-4393-a12a-f06477f82928","Type":"ContainerStarted","Data":"ff8f960470de3e17d2024b7e005f3684d47372801be2297c4faddcf4bde6a0b8"} Oct 30 22:24:30.588283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:30.588172 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-hfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:24:30.588546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:30.588502 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podUID="82a49049-a92f-4eeb-aa52-ff15da31e664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 22:24:30.820946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:30.820917 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:30.822340 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:30.822319 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-m55sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:24:30.822462 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:30.822382 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 22:24:31.586839 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:31.586799 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-hfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:24:31.587263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:31.586850 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podUID="82a49049-a92f-4eeb-aa52-ff15da31e664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 22:24:31.821424 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:31.821388 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-m55sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:24:31.821605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:31.821455 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 22:24:32.586745 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:32.586708 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-hfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:24:32.586956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:32.586771 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podUID="82a49049-a92f-4eeb-aa52-ff15da31e664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 22:24:32.821450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:32.821367 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-m55sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:24:32.821450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:32.821428 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 22:24:33.586713 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:33.586679 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-hfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:24:33.586915 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:33.586734 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podUID="82a49049-a92f-4eeb-aa52-ff15da31e664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 22:24:33.821980 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:33.821936 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-m55sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:24:33.822409 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:33.822003 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 22:24:34.586883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:34.586844 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-hfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:24:34.587095 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:34.586923 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podUID="82a49049-a92f-4eeb-aa52-ff15da31e664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 22:24:34.821688 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:34.821646 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-m55sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:24:34.821876 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:34.821699 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 22:24:35.587070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:35.587032 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-hfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:24:35.587467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:35.587094 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podUID="82a49049-a92f-4eeb-aa52-ff15da31e664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 22:24:35.821565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:35.821528 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-m55sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:24:35.821760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:35.821579 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 22:24:36.586168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:36.586134 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-hfhbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:24:36.586337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:36.586185 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podUID="82a49049-a92f-4eeb-aa52-ff15da31e664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 22:24:36.821728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:36.821686 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-m55sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:24:36.822131 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:36.821754 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 22:24:37.815864 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:37.815829 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:37.816723 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:37.816695 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:37.817602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:37.817579 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" Oct 30 22:24:37.976135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:37.976080 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podStartSLOduration=9.976065632 podStartE2EDuration="9.976065632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:30.484932454 +0000 UTC m=+1361.742274165" watchObservedRunningTime="2026-10-30 22:24:37.976065632 +0000 UTC m=+1369.233407341" Oct 30 22:24:38.067133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:38.067041 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:38.067536 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:38.067513 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:38.068556 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:38.068531 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:24:45.755795 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:45.755648 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:24:54.412187 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:24:54.412147 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:24:56.816228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.815321 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29"] Oct 30 22:24:56.852235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.832897 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.870621 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.870574 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.870895 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.870871 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.871026 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.871010 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.871184 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.871170 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.871330 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.871314 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf81c2-95e4-43b0-999a-7f5620126a8d-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.871536 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.871519 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.871658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.871644 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.871797 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.871766 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.872019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.871994 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xh4\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-kube-api-access-56xh4\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.884762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.884735 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-05e3625f-openshift-default-dockercfg-8l4lr\"" Oct 30 22:24:56.943949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.943906 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29"] Oct 30 22:24:56.974033 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.973996 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.974222 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.974122 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.974222 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.974165 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf81c2-95e4-43b0-999a-7f5620126a8d-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.974393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.974365 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.974481 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.974428 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.974481 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.974471 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.974592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.974501 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xh4\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-kube-api-access-56xh4\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.974839 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.974813 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.974987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.974897 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.975155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.975135 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf81c2-95e4-43b0-999a-7f5620126a8d-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.975433 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.975409 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.975573 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.975555 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.975918 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.975881 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.976015 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.975993 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.978029 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.977998 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:56.978435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:56.978415 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:57.033473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:57.033438 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:57.036943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:57.036903 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xh4\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-kube-api-access-56xh4\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:57.158286 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:57.158173 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:57.604249 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:57.604214 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29"] Oct 30 22:24:57.608292 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:24:57.608265 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baf81c2_95e4_43b0_999a_7f5620126a8d.slice/crio-62329cddc872afb504cfba02c7f809759fb6fe39d1b5d8cf0d3fb03a48997b76 WatchSource:0}: Error finding container 62329cddc872afb504cfba02c7f809759fb6fe39d1b5d8cf0d3fb03a48997b76: Status 404 returned error can't find the container with id 62329cddc872afb504cfba02c7f809759fb6fe39d1b5d8cf0d3fb03a48997b76 Oct 30 22:24:57.610850 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:57.610815 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:57.610946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:57.610886 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:57.610946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:57.610916 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:24:58.492490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:58.492456 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" event={"ID":"0baf81c2-95e4-43b0-999a-7f5620126a8d","Type":"ContainerStarted","Data":"5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0"} Oct 30 22:24:58.492906 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:58.492515 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" event={"ID":"0baf81c2-95e4-43b0-999a-7f5620126a8d","Type":"ContainerStarted","Data":"62329cddc872afb504cfba02c7f809759fb6fe39d1b5d8cf0d3fb03a48997b76"} Oct 30 22:24:58.661939 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:58.661881 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" podStartSLOduration=2.661868172 podStartE2EDuration="2.661868172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:58.659304383 +0000 UTC m=+1389.916646101" watchObservedRunningTime="2026-10-30 22:24:58.661868172 +0000 UTC m=+1389.919209882" Oct 30 22:24:59.158967 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:59.158929 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:24:59.160559 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:59.160521 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:24:59.160706 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:24:59.160584 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 22:25:00.159083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:00.159047 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:25:00.159532 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:00.159127 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 22:25:01.158567 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:01.158534 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:25:01.158762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:01.158585 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 22:25:02.160087 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:02.160052 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:25:02.160521 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:02.160109 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 22:25:03.158736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:03.158688 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:25:03.158938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:03.158750 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 22:25:04.163040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:04.163010 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:25:04.163488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:04.163468 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:25:04.164191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:04.164169 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:25:14.491592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:14.491556 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh"] Oct 30 22:25:14.592107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:14.592071 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz"] Oct 30 22:25:19.492802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.492735 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" podUID="d0aa45f6-f151-4d93-b26e-99e6423c05cc" containerName="payload-processing" containerID="cri-o://1cd90441d7f45ea69c61be241ce701d9e262bb3a3e8b9df7b23cc18dca59e111" gracePeriod=25 Oct 30 22:25:19.592773 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.592699 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" podUID="255db097-fbbc-4d21-823b-d4dd796c30df" containerName="payload-pre-processing" containerID="cri-o://19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300" gracePeriod=25 Oct 30 22:25:19.680882 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.680768 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" event={"ID":"d0aa45f6-f151-4d93-b26e-99e6423c05cc","Type":"ContainerDied","Data":"1cd90441d7f45ea69c61be241ce701d9e262bb3a3e8b9df7b23cc18dca59e111"} Oct 30 22:25:19.681109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.680635 5226 generic.go:356] "Generic (PLEG): container finished" podID="d0aa45f6-f151-4d93-b26e-99e6423c05cc" containerID="1cd90441d7f45ea69c61be241ce701d9e262bb3a3e8b9df7b23cc18dca59e111" exitCode=0 Oct 30 22:25:19.801750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.801715 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:25:19.864179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.864147 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0aa45f6-f151-4d93-b26e-99e6423c05cc-kube-api-access-pmdjh\" (UniqueName: \"kubernetes.io/projected/d0aa45f6-f151-4d93-b26e-99e6423c05cc-kube-api-access-pmdjh\") pod \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\" (UID: \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\") " Oct 30 22:25:19.864380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.864189 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0aa45f6-f151-4d93-b26e-99e6423c05cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aa45f6-f151-4d93-b26e-99e6423c05cc-plugins-config-volume\") pod \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\" (UID: \"d0aa45f6-f151-4d93-b26e-99e6423c05cc\") " Oct 30 22:25:19.864664 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.864641 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0aa45f6-f151-4d93-b26e-99e6423c05cc-plugins-config-volume" pod "d0aa45f6-f151-4d93-b26e-99e6423c05cc" (UID: "d0aa45f6-f151-4d93-b26e-99e6423c05cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:19.864908 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.864890 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0aa45f6-f151-4d93-b26e-99e6423c05cc-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:19.866412 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.866388 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0aa45f6-f151-4d93-b26e-99e6423c05cc-kube-api-access-pmdjh" pod "d0aa45f6-f151-4d93-b26e-99e6423c05cc" (UID: "d0aa45f6-f151-4d93-b26e-99e6423c05cc"). InnerVolumeSpecName "kube-api-access-pmdjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:19.891141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.891118 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:25:19.965860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.965810 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/255db097-fbbc-4d21-823b-d4dd796c30df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255db097-fbbc-4d21-823b-d4dd796c30df-plugins-config-volume\") pod \"255db097-fbbc-4d21-823b-d4dd796c30df\" (UID: \"255db097-fbbc-4d21-823b-d4dd796c30df\") " Oct 30 22:25:19.966055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.965914 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255db097-fbbc-4d21-823b-d4dd796c30df-kube-api-access-mb9dr\" (UniqueName: \"kubernetes.io/projected/255db097-fbbc-4d21-823b-d4dd796c30df-kube-api-access-mb9dr\") pod \"255db097-fbbc-4d21-823b-d4dd796c30df\" (UID: \"255db097-fbbc-4d21-823b-d4dd796c30df\") " Oct 30 22:25:19.966343 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.966310 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255db097-fbbc-4d21-823b-d4dd796c30df-plugins-config-volume" pod "255db097-fbbc-4d21-823b-d4dd796c30df" (UID: "255db097-fbbc-4d21-823b-d4dd796c30df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:19.966762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.966739 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmdjh\" (UniqueName: \"kubernetes.io/projected/d0aa45f6-f151-4d93-b26e-99e6423c05cc-kube-api-access-pmdjh\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:19.966868 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.966768 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/255db097-fbbc-4d21-823b-d4dd796c30df-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:19.968650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:19.968629 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255db097-fbbc-4d21-823b-d4dd796c30df-kube-api-access-mb9dr" pod "255db097-fbbc-4d21-823b-d4dd796c30df" (UID: "255db097-fbbc-4d21-823b-d4dd796c30df"). InnerVolumeSpecName "kube-api-access-mb9dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:20.068360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.068274 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9dr\" (UniqueName: \"kubernetes.io/projected/255db097-fbbc-4d21-823b-d4dd796c30df-kube-api-access-mb9dr\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:20.690454 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.690422 5226 generic.go:356] "Generic (PLEG): container finished" podID="255db097-fbbc-4d21-823b-d4dd796c30df" containerID="19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300" exitCode=0 Oct 30 22:25:20.690861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.690470 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" Oct 30 22:25:20.690861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.690509 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" event={"ID":"255db097-fbbc-4d21-823b-d4dd796c30df","Type":"ContainerDied","Data":"19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300"} Oct 30 22:25:20.690861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.690562 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz" event={"ID":"255db097-fbbc-4d21-823b-d4dd796c30df","Type":"ContainerDied","Data":"fff892223318638062af20e700dbbe98ba50a5bc679fa086ecbcd1acc4a80ed8"} Oct 30 22:25:20.690861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.690584 5226 scope.go:122] "RemoveContainer" containerID="19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300" Oct 30 22:25:20.692186 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.692159 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" event={"ID":"d0aa45f6-f151-4d93-b26e-99e6423c05cc","Type":"ContainerDied","Data":"d2cf2247c4ddb9c504bdd4b6397eab22e85ef4fbe586e3a0163ba011f971f35f"} Oct 30 22:25:20.692329 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.692237 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh" Oct 30 22:25:20.703067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.703051 5226 scope.go:122] "RemoveContainer" containerID="19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300" Oct 30 22:25:20.703320 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:20.703298 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300\": container with ID starting with 19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300 not found: ID does not exist" containerID="19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300" Oct 30 22:25:20.703420 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.703326 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300"} err="failed to get container status \"19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300\": rpc error: code = NotFound desc = could not find container \"19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300\": container with ID starting with 19bf1f0a39abc703595a0590ade5ce012d9c859d6a501bf192990fc70643f300 not found: ID does not exist" Oct 30 22:25:20.703420 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.703342 5226 scope.go:122] "RemoveContainer" containerID="1cd90441d7f45ea69c61be241ce701d9e262bb3a3e8b9df7b23cc18dca59e111" Oct 30 22:25:20.845900 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.845758 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz"] Oct 30 22:25:20.891811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.891777 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-05e3625f-7488d6cbc5-gnzwz"] Oct 30 22:25:20.907784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.907755 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="255db097-fbbc-4d21-823b-d4dd796c30df" path="/var/lib/kubelet/pods/255db097-fbbc-4d21-823b-d4dd796c30df/volumes" Oct 30 22:25:20.995003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:20.994969 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh"] Oct 30 22:25:21.001469 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:21.001440 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-05e3625f-5c977df99-tpplh"] Oct 30 22:25:22.906490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:22.906455 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0aa45f6-f151-4d93-b26e-99e6423c05cc" path="/var/lib/kubelet/pods/d0aa45f6-f151-4d93-b26e-99e6423c05cc/volumes" Oct 30 22:25:24.411804 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:24.411767 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:25:25.435562 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:25.435527 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs"] Oct 30 22:25:25.534188 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:25.534156 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm"] Oct 30 22:25:30.436933 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.436884 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" podUID="d10a43fd-f5a8-4488-8c02-c9adb876a639" containerName="payload-processing" containerID="cri-o://da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed" gracePeriod=25 Oct 30 22:25:30.535041 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.534998 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" podUID="b072daa7-79bd-4374-90fe-50b99315006d" containerName="payload-pre-processing" containerID="cri-o://226cf36a8182aad47c6f4838f8cdb41ec2ecf8fe7b6d0521e4d5d0ad8a631529" gracePeriod=25 Oct 30 22:25:30.739806 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.739431 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:25:30.774883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.774834 5226 generic.go:356] "Generic (PLEG): container finished" podID="b072daa7-79bd-4374-90fe-50b99315006d" containerID="226cf36a8182aad47c6f4838f8cdb41ec2ecf8fe7b6d0521e4d5d0ad8a631529" exitCode=0 Oct 30 22:25:30.775075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.774996 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" event={"ID":"b072daa7-79bd-4374-90fe-50b99315006d","Type":"ContainerDied","Data":"226cf36a8182aad47c6f4838f8cdb41ec2ecf8fe7b6d0521e4d5d0ad8a631529"} Oct 30 22:25:30.776803 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.776774 5226 generic.go:356] "Generic (PLEG): container finished" podID="d10a43fd-f5a8-4488-8c02-c9adb876a639" containerID="da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed" exitCode=0 Oct 30 22:25:30.776938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.776816 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" event={"ID":"d10a43fd-f5a8-4488-8c02-c9adb876a639","Type":"ContainerDied","Data":"da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed"} Oct 30 22:25:30.776938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.776836 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" Oct 30 22:25:30.776938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.776863 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs" event={"ID":"d10a43fd-f5a8-4488-8c02-c9adb876a639","Type":"ContainerDied","Data":"ed36ecebe83c4a920c91e2667c3763ce438360f1178a66e8d383d550b1ba5534"} Oct 30 22:25:30.776938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.776895 5226 scope.go:122] "RemoveContainer" containerID="da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed" Oct 30 22:25:30.789973 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.789950 5226 scope.go:122] "RemoveContainer" containerID="da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed" Oct 30 22:25:30.790387 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:30.790360 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed\": container with ID starting with da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed not found: ID does not exist" containerID="da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed" Oct 30 22:25:30.790502 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.790395 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed"} err="failed to get container status \"da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed\": rpc error: code = NotFound desc = could not find container \"da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed\": container with ID starting with da04d8eab5e887467e669b48988c355bf7429745df23a5f31eabb4f1424912ed not found: ID does not exist" Oct 30 22:25:30.821619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.821592 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:25:30.848491 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.848461 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10a43fd-f5a8-4488-8c02-c9adb876a639-kube-api-access-c2gk4\" (UniqueName: \"kubernetes.io/projected/d10a43fd-f5a8-4488-8c02-c9adb876a639-kube-api-access-c2gk4\") pod \"d10a43fd-f5a8-4488-8c02-c9adb876a639\" (UID: \"d10a43fd-f5a8-4488-8c02-c9adb876a639\") " Oct 30 22:25:30.848661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.848601 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10a43fd-f5a8-4488-8c02-c9adb876a639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10a43fd-f5a8-4488-8c02-c9adb876a639-plugins-config-volume\") pod \"d10a43fd-f5a8-4488-8c02-c9adb876a639\" (UID: \"d10a43fd-f5a8-4488-8c02-c9adb876a639\") " Oct 30 22:25:30.849285 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.849247 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10a43fd-f5a8-4488-8c02-c9adb876a639-plugins-config-volume" pod "d10a43fd-f5a8-4488-8c02-c9adb876a639" (UID: "d10a43fd-f5a8-4488-8c02-c9adb876a639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:30.850720 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.850700 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10a43fd-f5a8-4488-8c02-c9adb876a639-kube-api-access-c2gk4" pod "d10a43fd-f5a8-4488-8c02-c9adb876a639" (UID: "d10a43fd-f5a8-4488-8c02-c9adb876a639"). InnerVolumeSpecName "kube-api-access-c2gk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:30.950251 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.950166 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b072daa7-79bd-4374-90fe-50b99315006d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072daa7-79bd-4374-90fe-50b99315006d-plugins-config-volume\") pod \"b072daa7-79bd-4374-90fe-50b99315006d\" (UID: \"b072daa7-79bd-4374-90fe-50b99315006d\") " Oct 30 22:25:30.950397 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.950307 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b072daa7-79bd-4374-90fe-50b99315006d-kube-api-access-vljw2\" (UniqueName: \"kubernetes.io/projected/b072daa7-79bd-4374-90fe-50b99315006d-kube-api-access-vljw2\") pod \"b072daa7-79bd-4374-90fe-50b99315006d\" (UID: \"b072daa7-79bd-4374-90fe-50b99315006d\") " Oct 30 22:25:30.950623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.950599 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b072daa7-79bd-4374-90fe-50b99315006d-plugins-config-volume" pod "b072daa7-79bd-4374-90fe-50b99315006d" (UID: "b072daa7-79bd-4374-90fe-50b99315006d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:30.951041 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.951015 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2gk4\" (UniqueName: \"kubernetes.io/projected/d10a43fd-f5a8-4488-8c02-c9adb876a639-kube-api-access-c2gk4\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:30.951127 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.951043 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072daa7-79bd-4374-90fe-50b99315006d-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:30.951127 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.951057 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d10a43fd-f5a8-4488-8c02-c9adb876a639-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:30.952328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:30.952309 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b072daa7-79bd-4374-90fe-50b99315006d-kube-api-access-vljw2" pod "b072daa7-79bd-4374-90fe-50b99315006d" (UID: "b072daa7-79bd-4374-90fe-50b99315006d"). InnerVolumeSpecName "kube-api-access-vljw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:31.052327 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:31.052289 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vljw2\" (UniqueName: \"kubernetes.io/projected/b072daa7-79bd-4374-90fe-50b99315006d-kube-api-access-vljw2\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:31.185340 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:31.185305 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs"] Oct 30 22:25:31.190553 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:31.190522 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae20a62d-76cbb85fc-2j2qs"] Oct 30 22:25:31.793395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:31.793290 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" Oct 30 22:25:31.793395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:31.793314 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm" event={"ID":"b072daa7-79bd-4374-90fe-50b99315006d","Type":"ContainerDied","Data":"824c132727b6ae5437e0495176c3a201c0a792f4138d0f580167032903aeea15"} Oct 30 22:25:31.793908 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:31.793413 5226 scope.go:122] "RemoveContainer" containerID="226cf36a8182aad47c6f4838f8cdb41ec2ecf8fe7b6d0521e4d5d0ad8a631529" Oct 30 22:25:31.952312 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:31.952276 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm"] Oct 30 22:25:31.998486 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:31.998451 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae20a62d-5c764c66f5-68fxm"] Oct 30 22:25:32.911834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:32.911797 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b072daa7-79bd-4374-90fe-50b99315006d" path="/var/lib/kubelet/pods/b072daa7-79bd-4374-90fe-50b99315006d/volumes" Oct 30 22:25:32.912498 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:32.912470 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10a43fd-f5a8-4488-8c02-c9adb876a639" path="/var/lib/kubelet/pods/d10a43fd-f5a8-4488-8c02-c9adb876a639/volumes" Oct 30 22:25:34.854325 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:34.854291 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg"] Oct 30 22:25:34.950330 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:34.950295 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv"] Oct 30 22:25:39.858937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:39.858865 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" podUID="1f3d5470-9294-41dd-8998-0663648e4197" containerName="payload-processing" containerID="cri-o://86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c" gracePeriod=25 Oct 30 22:25:39.951523 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:39.951476 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" podUID="b4f65b40-ed93-4cbf-8dc9-03327d473926" containerName="payload-pre-processing" containerID="cri-o://131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3" gracePeriod=25 Oct 30 22:25:40.231966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.231935 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:25:40.259935 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.259897 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3d5470-9294-41dd-8998-0663648e4197-kube-api-access-89pjk\" (UniqueName: \"kubernetes.io/projected/1f3d5470-9294-41dd-8998-0663648e4197-kube-api-access-89pjk\") pod \"1f3d5470-9294-41dd-8998-0663648e4197\" (UID: \"1f3d5470-9294-41dd-8998-0663648e4197\") " Oct 30 22:25:40.260195 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.260176 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3d5470-9294-41dd-8998-0663648e4197-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d5470-9294-41dd-8998-0663648e4197-plugins-config-volume\") pod \"1f3d5470-9294-41dd-8998-0663648e4197\" (UID: \"1f3d5470-9294-41dd-8998-0663648e4197\") " Oct 30 22:25:40.260714 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.260684 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3d5470-9294-41dd-8998-0663648e4197-plugins-config-volume" pod "1f3d5470-9294-41dd-8998-0663648e4197" (UID: "1f3d5470-9294-41dd-8998-0663648e4197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:40.260980 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.260948 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3d5470-9294-41dd-8998-0663648e4197-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.261063 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.261025 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:25:40.262281 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.262259 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3d5470-9294-41dd-8998-0663648e4197-kube-api-access-89pjk" pod "1f3d5470-9294-41dd-8998-0663648e4197" (UID: "1f3d5470-9294-41dd-8998-0663648e4197"). InnerVolumeSpecName "kube-api-access-89pjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:40.361836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.361800 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f65b40-ed93-4cbf-8dc9-03327d473926-kube-api-access-62svv\" (UniqueName: \"kubernetes.io/projected/b4f65b40-ed93-4cbf-8dc9-03327d473926-kube-api-access-62svv\") pod \"b4f65b40-ed93-4cbf-8dc9-03327d473926\" (UID: \"b4f65b40-ed93-4cbf-8dc9-03327d473926\") " Oct 30 22:25:40.362044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.361895 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f65b40-ed93-4cbf-8dc9-03327d473926-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f65b40-ed93-4cbf-8dc9-03327d473926-plugins-config-volume\") pod \"b4f65b40-ed93-4cbf-8dc9-03327d473926\" (UID: \"b4f65b40-ed93-4cbf-8dc9-03327d473926\") " Oct 30 22:25:40.362586 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.362546 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f65b40-ed93-4cbf-8dc9-03327d473926-plugins-config-volume" pod "b4f65b40-ed93-4cbf-8dc9-03327d473926" (UID: "b4f65b40-ed93-4cbf-8dc9-03327d473926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:40.362791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.362773 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f65b40-ed93-4cbf-8dc9-03327d473926-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.362862 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.362792 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89pjk\" (UniqueName: \"kubernetes.io/projected/1f3d5470-9294-41dd-8998-0663648e4197-kube-api-access-89pjk\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.364256 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.364192 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f65b40-ed93-4cbf-8dc9-03327d473926-kube-api-access-62svv" pod "b4f65b40-ed93-4cbf-8dc9-03327d473926" (UID: "b4f65b40-ed93-4cbf-8dc9-03327d473926"). InnerVolumeSpecName "kube-api-access-62svv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:40.464888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.464850 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62svv\" (UniqueName: \"kubernetes.io/projected/b4f65b40-ed93-4cbf-8dc9-03327d473926-kube-api-access-62svv\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:40.859184 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.859093 5226 generic.go:356] "Generic (PLEG): container finished" podID="b4f65b40-ed93-4cbf-8dc9-03327d473926" containerID="131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3" exitCode=0 Oct 30 22:25:40.859184 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.859142 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" Oct 30 22:25:40.859709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.859176 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" event={"ID":"b4f65b40-ed93-4cbf-8dc9-03327d473926","Type":"ContainerDied","Data":"131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3"} Oct 30 22:25:40.859709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.859290 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv" event={"ID":"b4f65b40-ed93-4cbf-8dc9-03327d473926","Type":"ContainerDied","Data":"6b7ed81eedba8e6322d2489f4a1bc3cb4443b4d578439b7e6bbd389e59ef6fcd"} Oct 30 22:25:40.859709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.859326 5226 scope.go:122] "RemoveContainer" containerID="131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3" Oct 30 22:25:40.860822 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.860795 5226 generic.go:356] "Generic (PLEG): container finished" podID="1f3d5470-9294-41dd-8998-0663648e4197" containerID="86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c" exitCode=0 Oct 30 22:25:40.860923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.860840 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" event={"ID":"1f3d5470-9294-41dd-8998-0663648e4197","Type":"ContainerDied","Data":"86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c"} Oct 30 22:25:40.860923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.860882 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" event={"ID":"1f3d5470-9294-41dd-8998-0663648e4197","Type":"ContainerDied","Data":"be689265059b9994722df0afc3a705a2ec29946781e05d87c330f9106af60e12"} Oct 30 22:25:40.860923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.860843 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg" Oct 30 22:25:40.869504 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.869486 5226 scope.go:122] "RemoveContainer" containerID="131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3" Oct 30 22:25:40.869817 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:40.869797 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3\": container with ID starting with 131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3 not found: ID does not exist" containerID="131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3" Oct 30 22:25:40.869877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.869825 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3"} err="failed to get container status \"131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3\": rpc error: code = NotFound desc = could not find container \"131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3\": container with ID starting with 131c3d1a32a47dcd9d51a209ffd0746a4122ac19a4e430865aabb62a083a9ab3 not found: ID does not exist" Oct 30 22:25:40.869877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.869842 5226 scope.go:122] "RemoveContainer" containerID="86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c" Oct 30 22:25:40.884334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.884313 5226 scope.go:122] "RemoveContainer" containerID="86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c" Oct 30 22:25:40.884673 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:40.884649 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c\": container with ID starting with 86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c not found: ID does not exist" containerID="86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c" Oct 30 22:25:40.884731 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:40.884682 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c"} err="failed to get container status \"86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c\": rpc error: code = NotFound desc = could not find container \"86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c\": container with ID starting with 86519a58d2aa98c3a84152afbebc860203b921796e8b107418c13966a52a356c not found: ID does not exist" Oct 30 22:25:41.014736 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.014699 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg"] Oct 30 22:25:41.061018 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.060986 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dee83803-7f495bcb9f-xqbzg"] Oct 30 22:25:41.121567 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.121478 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv"] Oct 30 22:25:41.124826 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.124800 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dee83803-cdc57876c-xfwmv"] Oct 30 22:25:41.800171 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800137 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz"] Oct 30 22:25:41.800828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800802 5226 cpu_manager.go:414] "containerMap: removing container" podUID="b4f65b40-ed93-4cbf-8dc9-03327d473926" containerName="payload-pre-processing" Oct 30 22:25:41.800828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800828 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f65b40-ed93-4cbf-8dc9-03327d473926" containerName="payload-pre-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800856 5226 cpu_manager.go:414] "containerMap: removing container" podUID="255db097-fbbc-4d21-823b-d4dd796c30df" containerName="payload-pre-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800864 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255db097-fbbc-4d21-823b-d4dd796c30df" containerName="payload-pre-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800879 5226 cpu_manager.go:414] "containerMap: removing container" podUID="d0aa45f6-f151-4d93-b26e-99e6423c05cc" containerName="payload-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800886 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0aa45f6-f151-4d93-b26e-99e6423c05cc" containerName="payload-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800911 5226 cpu_manager.go:414] "containerMap: removing container" podUID="d10a43fd-f5a8-4488-8c02-c9adb876a639" containerName="payload-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800917 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10a43fd-f5a8-4488-8c02-c9adb876a639" containerName="payload-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800929 5226 cpu_manager.go:414] "containerMap: removing container" podUID="1f3d5470-9294-41dd-8998-0663648e4197" containerName="payload-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800935 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3d5470-9294-41dd-8998-0663648e4197" containerName="payload-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800981 5226 cpu_manager.go:414] "containerMap: removing container" podUID="b072daa7-79bd-4374-90fe-50b99315006d" containerName="payload-pre-processing" Oct 30 22:25:41.801091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.800989 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b072daa7-79bd-4374-90fe-50b99315006d" containerName="payload-pre-processing" Oct 30 22:25:41.801716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.801142 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="b072daa7-79bd-4374-90fe-50b99315006d" containerName="payload-pre-processing" Oct 30 22:25:41.801716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.801157 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f65b40-ed93-4cbf-8dc9-03327d473926" containerName="payload-pre-processing" Oct 30 22:25:41.801716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.801169 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="255db097-fbbc-4d21-823b-d4dd796c30df" containerName="payload-pre-processing" Oct 30 22:25:41.801716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.801180 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0aa45f6-f151-4d93-b26e-99e6423c05cc" containerName="payload-processing" Oct 30 22:25:41.801716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.801191 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10a43fd-f5a8-4488-8c02-c9adb876a639" containerName="payload-processing" Oct 30 22:25:41.801716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.801218 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3d5470-9294-41dd-8998-0663648e4197" containerName="payload-processing" Oct 30 22:25:41.806413 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.806394 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.883512 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.883473 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ee812b-7eac-469d-ae3c-680804aa4264-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.884028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.883602 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.884028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.883681 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ee812b-7eac-469d-ae3c-680804aa4264-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.884028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.883811 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxmk\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-kube-api-access-gwxmk\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.884028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.883860 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.884028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.883981 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.884385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.884050 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.884385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.884083 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.884385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.884117 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.907899 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.907864 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz"] Oct 30 22:25:41.985563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.985525 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ee812b-7eac-469d-ae3c-680804aa4264-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.985775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.985635 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.985775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.985694 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ee812b-7eac-469d-ae3c-680804aa4264-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.985893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.985779 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxmk\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-kube-api-access-gwxmk\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.985893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.985825 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.985921 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.985986 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986101 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.986021 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986101 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.986056 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.986103 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.986255 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-certs\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986423 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.986392 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-data\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986547 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.986403 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-credential-socket\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.986547 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.986438 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ee812b-7eac-469d-ae3c-680804aa4264-istiod-ca-cert\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.988379 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.988354 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-envoy\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:41.988627 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:41.988610 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ee812b-7eac-469d-ae3c-680804aa4264-istio-podinfo\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:42.043693 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.043659 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-istio-token\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:42.043866 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.043843 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxmk\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-kube-api-access-gwxmk\") pod \"e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:42.120021 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.119942 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:42.556168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.556108 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz"] Oct 30 22:25:42.560690 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:25:42.560664 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ee812b_7eac_469d_ae3c_680804aa4264.slice/crio-75780db4454ea2b1900009ceb09e9dea637c824b2ce22084835ea7fdece9c743 WatchSource:0}: Error finding container 75780db4454ea2b1900009ceb09e9dea637c824b2ce22084835ea7fdece9c743: Status 404 returned error can't find the container with id 75780db4454ea2b1900009ceb09e9dea637c824b2ce22084835ea7fdece9c743 Oct 30 22:25:42.563320 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.563289 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:42.563372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.563357 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:42.563413 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.563386 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:42.882244 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.882151 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" event={"ID":"00ee812b-7eac-469d-ae3c-680804aa4264","Type":"ContainerStarted","Data":"8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30"} Oct 30 22:25:42.882244 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.882231 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" event={"ID":"00ee812b-7eac-469d-ae3c-680804aa4264","Type":"ContainerStarted","Data":"75780db4454ea2b1900009ceb09e9dea637c824b2ce22084835ea7fdece9c743"} Oct 30 22:25:42.907512 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.907475 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3d5470-9294-41dd-8998-0663648e4197" path="/var/lib/kubelet/pods/1f3d5470-9294-41dd-8998-0663648e4197/volumes" Oct 30 22:25:42.908133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.908111 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f65b40-ed93-4cbf-8dc9-03327d473926" path="/var/lib/kubelet/pods/b4f65b40-ed93-4cbf-8dc9-03327d473926/volumes" Oct 30 22:25:42.998803 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:42.998753 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" podStartSLOduration=1.998738528 podStartE2EDuration="1.998738528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:42.996792573 +0000 UTC m=+1434.254134284" watchObservedRunningTime="2026-10-30 22:25:42.998738528 +0000 UTC m=+1434.256080240" Oct 30 22:25:43.120873 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:43.120838 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:43.122644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:43.122609 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:25:43.122747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:43.122655 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:25:44.120875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:44.120840 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:25:44.121319 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:44.120894 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:25:45.120346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:45.120310 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:25:45.120557 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:45.120383 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:25:46.121218 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:46.121173 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:25:46.121623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:46.121250 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:25:47.121614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:47.120611 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:25:47.121614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:47.120714 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:25:48.120951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:48.120916 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 22:25:48.121138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:48.120979 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 22:25:48.391281 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:48.390743 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29"] Oct 30 22:25:48.391755 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:48.391465 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" containerID="cri-o://5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0" gracePeriod=30 Oct 30 22:25:48.406928 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:48.406893 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz"] Oct 30 22:25:48.407155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:48.407130 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" containerID="cri-o://8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30" gracePeriod=30 Oct 30 22:25:49.166668 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:49.166633 5226 patch_prober.go:29] interesting pod/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:25:49.166814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:49.166697 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:25:49.969653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:49.969617 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc"] Oct 30 22:25:49.975700 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:49.975677 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023367 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023330 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-data\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023367 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023377 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-podinfo\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023518 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-socket\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023545 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-credential-socket\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023623 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023604 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp8v\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-kube-api-access-tzp8v\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023629 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-certs\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023730 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a257a-9a73-4c06-bb86-b00e10c032e5-istiod-ca-cert\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023762 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-envoy\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.023792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.023784 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-token\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.029467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.029413 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cc8c7d75-gw-openshift-default-dockercfg-8k8vq\"" Oct 30 22:25:50.079075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.079043 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc"] Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.136708 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-socket\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.136771 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-credential-socket\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.136839 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp8v\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-kube-api-access-tzp8v\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.136883 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-certs\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.137008 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a257a-9a73-4c06-bb86-b00e10c032e5-istiod-ca-cert\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.137054 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-envoy\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.137088 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-token\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.137193 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-socket\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.137284 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-data\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.137312 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-podinfo\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.137800 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-credential-socket\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.138293 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-certs\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.140082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.138902 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a257a-9a73-4c06-bb86-b00e10c032e5-istiod-ca-cert\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.162242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.142523 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-data\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.169235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.168812 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-podinfo\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.196296 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.184284 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-envoy\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.290319 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.284290 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-token\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.290491 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.290454 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp8v\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-kube-api-access-tzp8v\") pod \"e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.297302 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.297268 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:50.751423 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.751388 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:50.751540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.751458 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:50.751540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.751485 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:50.767426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:50.767401 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc"] Oct 30 22:25:51.020229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:51.020102 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" event={"ID":"014a257a-9a73-4c06-bb86-b00e10c032e5","Type":"ContainerStarted","Data":"9871b32e1b17df8c640fe7d3a4b22d307870827b8987ce51a5780ada8f5deaa8"} Oct 30 22:25:51.020229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:51.020177 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" event={"ID":"014a257a-9a73-4c06-bb86-b00e10c032e5","Type":"ContainerStarted","Data":"d92e38725cad6cee5ecaec4018d4889400bcd21d692c3b5fc22a0710c3b26a7f"} Oct 30 22:25:51.135162 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:51.135096 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" podStartSLOduration=2.135079069 podStartE2EDuration="2.135079069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:51.134124012 +0000 UTC m=+1442.391465722" watchObservedRunningTime="2026-10-30 22:25:51.135079069 +0000 UTC m=+1442.392420779" Oct 30 22:25:51.298113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:51.298024 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:51.305563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:51.305533 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:52.027699 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:52.027665 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:52.028649 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:52.028626 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:25:53.722313 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.722279 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:25:53.832936 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.832901 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-socket\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.832947 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baf81c2-95e4-43b0-999a-7f5620126a8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf81c2-95e4-43b0-999a-7f5620126a8d-istiod-ca-cert\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.832978 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-kube-api-access-56xh4\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-kube-api-access-56xh4\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833036 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-data\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833106 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-podinfo\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833197 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-envoy\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833256 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-socket" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.833372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833306 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-certs\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833376 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-data" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.833522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833388 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-token\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-token\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833506 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-credential-socket\") pod \"0baf81c2-95e4-43b0-999a-7f5620126a8d\" (UID: \"0baf81c2-95e4-43b0-999a-7f5620126a8d\") " Oct 30 22:25:53.833522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833512 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-certs" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.833731 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833557 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baf81c2-95e4-43b0-999a-7f5620126a8d-istiod-ca-cert" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:53.833826 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.833798 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-credential-socket" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.834176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.834156 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.834282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.834180 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.834282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.834194 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf81c2-95e4-43b0-999a-7f5620126a8d-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.834282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.834252 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.834282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.834266 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.835404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.835376 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-podinfo" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:53.835629 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.835610 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-token" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:53.835817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.835797 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-envoy" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:53.835817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.835806 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-kube-api-access-56xh4" pod "0baf81c2-95e4-43b0-999a-7f5620126a8d" (UID: "0baf81c2-95e4-43b0-999a-7f5620126a8d"). InnerVolumeSpecName "kube-api-access-56xh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:53.935579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.935542 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.935579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.935573 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.935579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.935582 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xh4\" (UniqueName: \"kubernetes.io/projected/0baf81c2-95e4-43b0-999a-7f5620126a8d-kube-api-access-56xh4\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:53.935888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:53.935593 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf81c2-95e4-43b0-999a-7f5620126a8d-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.044055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.043966 5226 generic.go:356] "Generic (PLEG): container finished" podID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerID="5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0" exitCode=0 Oct 30 22:25:54.044055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.044017 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" Oct 30 22:25:54.044297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.044048 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" event={"ID":"0baf81c2-95e4-43b0-999a-7f5620126a8d","Type":"ContainerDied","Data":"5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0"} Oct 30 22:25:54.044297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.044099 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29" event={"ID":"0baf81c2-95e4-43b0-999a-7f5620126a8d","Type":"ContainerDied","Data":"62329cddc872afb504cfba02c7f809759fb6fe39d1b5d8cf0d3fb03a48997b76"} Oct 30 22:25:54.044297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.044125 5226 scope.go:122] "RemoveContainer" containerID="5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0" Oct 30 22:25:54.055698 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.055675 5226 scope.go:122] "RemoveContainer" containerID="5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0" Oct 30 22:25:54.055976 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:54.055957 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0\": container with ID starting with 5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0 not found: ID does not exist" containerID="5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0" Oct 30 22:25:54.056034 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.055984 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0"} err="failed to get container status \"5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0\": rpc error: code = NotFound desc = could not find container \"5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0\": container with ID starting with 5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0 not found: ID does not exist" Oct 30 22:25:54.184721 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:54.184678 5226 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baf81c2_95e4_43b0_999a_7f5620126a8d.slice/crio-conmon-5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ee812b_7eac_469d_ae3c_680804aa4264.slice/crio-conmon-8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baf81c2_95e4_43b0_999a_7f5620126a8d.slice/crio-5ec99bf66ef04c37fdd253dd0d6a217d50eaf2b4c69f8fefde0edcffc63414f0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baf81c2_95e4_43b0_999a_7f5620126a8d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ee812b_7eac_469d_ae3c_680804aa4264.slice/crio-8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:25:54.210570 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.210532 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29"] Oct 30 22:25:54.215344 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.215318 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-k5q29"] Oct 30 22:25:54.305036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.305007 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:54.412400 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:54.412361 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:25:54.443100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443064 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-envoy\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.443283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443263 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-data\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.443487 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443463 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-socket\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.443616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443502 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ee812b-7eac-469d-ae3c-680804aa4264-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ee812b-7eac-469d-ae3c-680804aa4264-istiod-ca-cert\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.443616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443518 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-data" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.443616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443604 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-certs\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.443788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443646 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-istio-token\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-istio-token\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.443788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443683 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00ee812b-7eac-469d-ae3c-680804aa4264-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ee812b-7eac-469d-ae3c-680804aa4264-istio-podinfo\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.443788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443716 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-credential-socket\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.443788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443751 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-kube-api-access-gwxmk\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-kube-api-access-gwxmk\") pod \"00ee812b-7eac-469d-ae3c-680804aa4264\" (UID: \"00ee812b-7eac-469d-ae3c-680804aa4264\") " Oct 30 22:25:54.444001 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.443859 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ee812b-7eac-469d-ae3c-680804aa4264-istiod-ca-cert" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:54.444231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.444179 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-credential-socket" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.444585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.444563 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ee812b-7eac-469d-ae3c-680804aa4264-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.444681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.444592 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.444681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.444607 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.444935 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.444912 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-certs" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.445015 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.444977 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-socket" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.446151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.446125 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-envoy" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:54.446533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.446506 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00ee812b-7eac-469d-ae3c-680804aa4264-istio-podinfo" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:54.446638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.446565 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-istio-token" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:54.447121 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.447100 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-kube-api-access-gwxmk" pod "00ee812b-7eac-469d-ae3c-680804aa4264" (UID: "00ee812b-7eac-469d-ae3c-680804aa4264"). InnerVolumeSpecName "kube-api-access-gwxmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:54.545984 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.545948 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.545984 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.545976 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.546181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.545985 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.546181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.546011 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ee812b-7eac-469d-ae3c-680804aa4264-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.546181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.546021 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwxmk\" (UniqueName: \"kubernetes.io/projected/00ee812b-7eac-469d-ae3c-680804aa4264-kube-api-access-gwxmk\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.546181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.546030 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ee812b-7eac-469d-ae3c-680804aa4264-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.908404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:54.908321 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" path="/var/lib/kubelet/pods/0baf81c2-95e4-43b0-999a-7f5620126a8d/volumes" Oct 30 22:25:55.053321 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.053286 5226 generic.go:356] "Generic (PLEG): container finished" podID="00ee812b-7eac-469d-ae3c-680804aa4264" containerID="8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30" exitCode=0 Oct 30 22:25:55.053503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.053345 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" Oct 30 22:25:55.053503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.053373 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" event={"ID":"00ee812b-7eac-469d-ae3c-680804aa4264","Type":"ContainerDied","Data":"8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30"} Oct 30 22:25:55.053503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.053431 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz" event={"ID":"00ee812b-7eac-469d-ae3c-680804aa4264","Type":"ContainerDied","Data":"75780db4454ea2b1900009ceb09e9dea637c824b2ce22084835ea7fdece9c743"} Oct 30 22:25:55.053503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.053454 5226 scope.go:122] "RemoveContainer" containerID="8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30" Oct 30 22:25:55.065244 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.065193 5226 scope.go:122] "RemoveContainer" containerID="8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30" Oct 30 22:25:55.065540 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:25:55.065520 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30\": container with ID starting with 8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30 not found: ID does not exist" containerID="8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30" Oct 30 22:25:55.065593 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.065548 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30"} err="failed to get container status \"8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30\": rpc error: code = NotFound desc = could not find container \"8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30\": container with ID starting with 8be7698b577cba25f42180116f0f258c0e6e3061504fcf1076751114379fbb30 not found: ID does not exist" Oct 30 22:25:55.211377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.211345 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz"] Oct 30 22:25:55.257595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:55.257557 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-05e3625f-openshift-default-7fbdbcf75c-4ggnz"] Oct 30 22:25:56.534402 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.534352 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk"] Oct 30 22:25:56.535271 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.535247 5226 cpu_manager.go:414] "containerMap: removing container" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" Oct 30 22:25:56.535271 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.535268 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" Oct 30 22:25:56.535450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.535283 5226 cpu_manager.go:414] "containerMap: removing container" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" Oct 30 22:25:56.535450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.535292 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" Oct 30 22:25:56.535450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.535442 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" containerName="istio-proxy" Oct 30 22:25:56.535544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.535457 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baf81c2-95e4-43b0-999a-7f5620126a8d" containerName="istio-proxy" Oct 30 22:25:56.540751 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.540725 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:56.575238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.575174 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4059d749-968e-4e37-a519-b166cb7418b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk\" (UID: \"4059d749-968e-4e37-a519-b166cb7418b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:56.575410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.575389 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7tl\" (UniqueName: \"kubernetes.io/projected/4059d749-968e-4e37-a519-b166cb7418b1-kube-api-access-dp7tl\") pod \"payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk\" (UID: \"4059d749-968e-4e37-a519-b166cb7418b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:56.602233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.597311 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk"] Oct 30 22:25:56.647148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.634910 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cc8c7d75-dockercfg-4zj8r\"" Oct 30 22:25:56.677958 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.677898 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7tl\" (UniqueName: \"kubernetes.io/projected/4059d749-968e-4e37-a519-b166cb7418b1-kube-api-access-dp7tl\") pod \"payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk\" (UID: \"4059d749-968e-4e37-a519-b166cb7418b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:56.678382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.678352 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4059d749-968e-4e37-a519-b166cb7418b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk\" (UID: \"4059d749-968e-4e37-a519-b166cb7418b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:56.678989 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.678965 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cc8c7d75\"" Oct 30 22:25:56.680137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.680112 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq"] Oct 30 22:25:56.685490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.685470 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:56.690535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.690508 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4059d749-968e-4e37-a519-b166cb7418b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk\" (UID: \"4059d749-968e-4e37-a519-b166cb7418b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:56.736669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.736639 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7tl\" (UniqueName: \"kubernetes.io/projected/4059d749-968e-4e37-a519-b166cb7418b1-kube-api-access-dp7tl\") pod \"payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk\" (UID: \"4059d749-968e-4e37-a519-b166cb7418b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:56.738509 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.738477 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq"] Oct 30 22:25:56.780597 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.780561 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5sx7\" (UniqueName: \"kubernetes.io/projected/0a283f99-0a43-4723-9826-481518aae650-kube-api-access-t5sx7\") pod \"payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq\" (UID: \"0a283f99-0a43-4723-9826-481518aae650\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:56.780968 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.780946 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a283f99-0a43-4723-9826-481518aae650-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq\" (UID: \"0a283f99-0a43-4723-9826-481518aae650\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:56.869103 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.868280 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:56.882749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.882715 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sx7\" (UniqueName: \"kubernetes.io/projected/0a283f99-0a43-4723-9826-481518aae650-kube-api-access-t5sx7\") pod \"payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq\" (UID: \"0a283f99-0a43-4723-9826-481518aae650\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:56.882894 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.882879 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a283f99-0a43-4723-9826-481518aae650-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq\" (UID: \"0a283f99-0a43-4723-9826-481518aae650\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:56.883914 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.883890 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a283f99-0a43-4723-9826-481518aae650-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq\" (UID: \"0a283f99-0a43-4723-9826-481518aae650\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:56.908749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.908718 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ee812b-7eac-469d-ae3c-680804aa4264" path="/var/lib/kubelet/pods/00ee812b-7eac-469d-ae3c-680804aa4264/volumes" Oct 30 22:25:56.953850 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:56.953819 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sx7\" (UniqueName: \"kubernetes.io/projected/0a283f99-0a43-4723-9826-481518aae650-kube-api-access-t5sx7\") pod \"payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq\" (UID: \"0a283f99-0a43-4723-9826-481518aae650\") " pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:57.004732 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:57.004697 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:57.274949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:57.274920 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk"] Oct 30 22:25:57.278857 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:25:57.278829 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4059d749_968e_4e37_a519_b166cb7418b1.slice/crio-446c099ad80f1da3d9742b0e83dcbe3d8e691e50300ef0b69717e10b64e5c2a5 WatchSource:0}: Error finding container 446c099ad80f1da3d9742b0e83dcbe3d8e691e50300ef0b69717e10b64e5c2a5: Status 404 returned error can't find the container with id 446c099ad80f1da3d9742b0e83dcbe3d8e691e50300ef0b69717e10b64e5c2a5 Oct 30 22:25:57.456521 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:57.456493 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq"] Oct 30 22:25:58.081618 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.081577 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" event={"ID":"4059d749-968e-4e37-a519-b166cb7418b1","Type":"ContainerStarted","Data":"3dc374d52e6dbea05db1213433d6c8f381a818396c6aec537db4a6e6aacec2c1"} Oct 30 22:25:58.082077 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.081642 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" event={"ID":"4059d749-968e-4e37-a519-b166cb7418b1","Type":"ContainerStarted","Data":"446c099ad80f1da3d9742b0e83dcbe3d8e691e50300ef0b69717e10b64e5c2a5"} Oct 30 22:25:58.082077 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.081799 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:25:58.083324 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.083294 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" event={"ID":"0a283f99-0a43-4723-9826-481518aae650","Type":"ContainerStarted","Data":"152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e"} Oct 30 22:25:58.083445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.083344 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" event={"ID":"0a283f99-0a43-4723-9826-481518aae650","Type":"ContainerStarted","Data":"1b171f90d8b9f7bba7238a7b5121b2c8e893eadf56e7c63f036ab9221298980b"} Oct 30 22:25:58.083445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.083375 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:25:58.192037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.191980 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" podStartSLOduration=2.191965316 podStartE2EDuration="2.191965316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:58.190047671 +0000 UTC m=+1449.447389381" watchObservedRunningTime="2026-10-30 22:25:58.191965316 +0000 UTC m=+1449.449307026" Oct 30 22:25:58.302860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.302805 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" podStartSLOduration=2.302790241 podStartE2EDuration="2.302790241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:58.30174419 +0000 UTC m=+1449.559085901" watchObservedRunningTime="2026-10-30 22:25:58.302790241 +0000 UTC m=+1449.560132008" Oct 30 22:25:58.925275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.925063 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp"] Oct 30 22:25:58.929868 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.929837 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956306 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb6098e-aac7-4b16-94fc-e14dd96de102-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956521 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956585 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956637 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9v2j\" (UniqueName: \"kubernetes.io/projected/eeb6098e-aac7-4b16-94fc-e14dd96de102-kube-api-access-m9v2j\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956718 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956824 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956857 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956903 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:58.957237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:58.956979 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.036934 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.036113 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp"] Oct 30 22:25:59.065624 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.065560 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb6098e-aac7-4b16-94fc-e14dd96de102-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.065949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.065930 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.066071 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.066059 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.066179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.066166 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9v2j\" (UniqueName: \"kubernetes.io/projected/eeb6098e-aac7-4b16-94fc-e14dd96de102-kube-api-access-m9v2j\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.066322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.066309 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.066493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.066481 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.066594 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.066580 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.066682 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.066671 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.067892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.067865 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb6098e-aac7-4b16-94fc-e14dd96de102-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.069515 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.069493 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.071114 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.071089 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.071511 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.071492 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.072143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.072122 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.073535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.073510 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.075369 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.075345 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.075673 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.075648 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.129230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.128731 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb6098e-aac7-4b16-94fc-e14dd96de102-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.131145 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.131103 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9v2j\" (UniqueName: \"kubernetes.io/projected/eeb6098e-aac7-4b16-94fc-e14dd96de102-kube-api-access-m9v2j\") pod \"maas-default-gateway-openshift-default-75bf867b6c-nr4sp\" (UID: \"eeb6098e-aac7-4b16-94fc-e14dd96de102\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.248663 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.248381 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:25:59.721199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.721152 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp"] Oct 30 22:25:59.744614 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.744566 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:59.744766 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.744661 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:25:59.744766 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:25:59.744708 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:00.113181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:00.113087 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" event={"ID":"eeb6098e-aac7-4b16-94fc-e14dd96de102","Type":"ContainerStarted","Data":"44aff6ded10a3c302e2ac74cc4c1ee0ce4417ecfa251c7ab78c9525f5cb643e8"} Oct 30 22:26:00.113181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:00.113157 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" event={"ID":"eeb6098e-aac7-4b16-94fc-e14dd96de102","Type":"ContainerStarted","Data":"e760fe6872bef8458836121e6670f5001e75071f54d9ecd0bd120bcd0a69728d"} Oct 30 22:26:00.245870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:00.245812 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podStartSLOduration=2.245796594 podStartE2EDuration="2.245796594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:00.2441089 +0000 UTC m=+1451.501450611" watchObservedRunningTime="2026-10-30 22:26:00.245796594 +0000 UTC m=+1451.503138303" Oct 30 22:26:00.248957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:00.248875 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:26:00.254149 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:00.254118 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:00.254386 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:00.254173 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:01.284240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:01.283818 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:01.284240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:01.283871 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:02.262999 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:02.262958 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:02.263311 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:02.263279 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:03.251649 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:03.251603 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:03.252178 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:03.251671 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:03.922147 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:03.922110 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:26:04.250234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:04.250111 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:04.250234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:04.250220 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:05.252683 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:05.252643 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:05.254487 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:05.254414 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:06.311587 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:06.311038 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:06.311587 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:06.311092 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:07.256230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:07.255953 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:07.256230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:07.256014 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:08.249685 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:08.249647 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 22:26:08.250243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:08.249709 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 22:26:09.096000 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:09.095970 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:26:09.096237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:09.096043 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:26:09.572791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:09.572759 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:26:09.573834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:09.573807 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:26:09.575359 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:09.575309 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" Oct 30 22:26:23.081113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.081077 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x"] Oct 30 22:26:23.085858 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.085835 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.135086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.135051 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:26:23.135086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.135061 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:26:23.135086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.135068 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:26:23.135365 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.135226 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7vbmp\"" Oct 30 22:26:23.185699 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.185659 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.185888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.185710 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.185963 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.185935 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.186077 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.186061 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5glp\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-kube-api-access-q5glp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.186117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.186099 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.186156 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.186126 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.186197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.186161 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/012c0719-751f-42c0-bb56-0776dae3569e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.191009 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.190976 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x"] Oct 30 22:26:23.288079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.288039 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.288305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.288114 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5glp\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-kube-api-access-q5glp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.288305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.288140 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.288305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.288156 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.288305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.288178 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/012c0719-751f-42c0-bb56-0776dae3569e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.288508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.288412 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.288508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.288432 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.288995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.288970 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.290799 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.290776 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/012c0719-751f-42c0-bb56-0776dae3569e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.290913 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.290808 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.291167 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.291145 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.291238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.291170 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.345588 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.345516 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.345719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.345690 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5glp\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-kube-api-access-q5glp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fxm8x\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.400330 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.400301 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:23.833715 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.833677 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x"] Oct 30 22:26:23.838395 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:26:23.838368 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012c0719_751f_42c0_bb56_0776dae3569e.slice/crio-170080382b5c406a48be9ef9c828d81a1bd8039954b4a4d47bd327ba45595b95 WatchSource:0}: Error finding container 170080382b5c406a48be9ef9c828d81a1bd8039954b4a4d47bd327ba45595b95: Status 404 returned error can't find the container with id 170080382b5c406a48be9ef9c828d81a1bd8039954b4a4d47bd327ba45595b95 Oct 30 22:26:23.840591 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:23.840572 5226 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:26:24.420675 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:26:24.420625 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:26:24.421286 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:24.420197 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" event={"ID":"012c0719-751f-42c0-bb56-0776dae3569e","Type":"ContainerStarted","Data":"170080382b5c406a48be9ef9c828d81a1bd8039954b4a4d47bd327ba45595b95"} Oct 30 22:26:28.122051 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:28.122015 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:28.122359 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:28.122085 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:26:28.453543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:28.453505 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" event={"ID":"012c0719-751f-42c0-bb56-0776dae3569e","Type":"ContainerStarted","Data":"a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4"} Oct 30 22:26:28.453718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:28.453642 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:28.663119 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:28.663062 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" podStartSLOduration=1.383097591 podStartE2EDuration="5.663046517s" totalImagesPullingTime="4.279948926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:28.614454745 +0000 UTC m=+1479.871796455" watchObservedRunningTime="2026-10-30 22:26:28.663046517 +0000 UTC m=+1479.920388227" Oct 30 22:26:29.465231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:29.465180 5226 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fxm8x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:29.465642 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:29.465302 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" podUID="012c0719-751f-42c0-bb56-0776dae3569e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:32.464144 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:32.464111 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:26:41.550538 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:41.550506 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:26:49.168307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:49.168269 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:26:49.174948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:26:49.174918 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:26:54.413648 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:26:54.413605 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:27:09.753711 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.753673 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws"] Oct 30 22:27:09.758442 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.758423 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:09.788836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.788805 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2056f2f9-5732-4c34-a715-5b7923c2cc93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws\" (UID: \"2056f2f9-5732-4c34-a715-5b7923c2cc93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:09.789051 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.789029 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dftth\" (UniqueName: \"kubernetes.io/projected/2056f2f9-5732-4c34-a715-5b7923c2cc93-kube-api-access-dftth\") pod \"payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws\" (UID: \"2056f2f9-5732-4c34-a715-5b7923c2cc93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:09.807041 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.807011 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-61346ee8\"" Oct 30 22:27:09.853463 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.853431 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-61346ee8-dockercfg-8nkd4\"" Oct 30 22:27:09.880064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.880031 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws"] Oct 30 22:27:09.890485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.890448 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2056f2f9-5732-4c34-a715-5b7923c2cc93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws\" (UID: \"2056f2f9-5732-4c34-a715-5b7923c2cc93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:09.891015 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.890986 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dftth\" (UniqueName: \"kubernetes.io/projected/2056f2f9-5732-4c34-a715-5b7923c2cc93-kube-api-access-dftth\") pod \"payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws\" (UID: \"2056f2f9-5732-4c34-a715-5b7923c2cc93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:09.896070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.892491 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2056f2f9-5732-4c34-a715-5b7923c2cc93-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws\" (UID: \"2056f2f9-5732-4c34-a715-5b7923c2cc93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:09.909451 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.909404 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q"] Oct 30 22:27:09.978828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.971762 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:09.980195 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.980161 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dftth\" (UniqueName: \"kubernetes.io/projected/2056f2f9-5732-4c34-a715-5b7923c2cc93-kube-api-access-dftth\") pod \"payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws\" (UID: \"2056f2f9-5732-4c34-a715-5b7923c2cc93\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:09.988238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:09.988199 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q"] Oct 30 22:27:10.032323 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.004504 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgn6x\" (UniqueName: \"kubernetes.io/projected/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-kube-api-access-bgn6x\") pod \"payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q\" (UID: \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\") " pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:10.032879 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.032852 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-plugins-config-volume\") pod \"payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q\" (UID: \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\") " pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:10.072656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.072628 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:10.134823 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.134789 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-plugins-config-volume\") pod \"payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q\" (UID: \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\") " pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:10.135070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.135050 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgn6x\" (UniqueName: \"kubernetes.io/projected/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-kube-api-access-bgn6x\") pod \"payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q\" (UID: \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\") " pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:10.135605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.135581 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-plugins-config-volume\") pod \"payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q\" (UID: \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\") " pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:10.197607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.197556 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgn6x\" (UniqueName: \"kubernetes.io/projected/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-kube-api-access-bgn6x\") pod \"payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q\" (UID: \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\") " pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:10.338738 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.338665 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:10.475421 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.475384 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws"] Oct 30 22:27:10.480647 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:27:10.480614 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2056f2f9_5732_4c34_a715_5b7923c2cc93.slice/crio-6c3db7db42ba1bb73b01474c5f25046e324c73657aa79c18aef26ed326007e74 WatchSource:0}: Error finding container 6c3db7db42ba1bb73b01474c5f25046e324c73657aa79c18aef26ed326007e74: Status 404 returned error can't find the container with id 6c3db7db42ba1bb73b01474c5f25046e324c73657aa79c18aef26ed326007e74 Oct 30 22:27:10.717660 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.717622 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q"] Oct 30 22:27:10.722295 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:27:10.722269 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee4c7ba_6ed4_475f_9e44_6fdcea8a7489.slice/crio-5d162952a75deb24f9bcdaad4310ae2de478d53f082d331ec8f43c93a60541e3 WatchSource:0}: Error finding container 5d162952a75deb24f9bcdaad4310ae2de478d53f082d331ec8f43c93a60541e3: Status 404 returned error can't find the container with id 5d162952a75deb24f9bcdaad4310ae2de478d53f082d331ec8f43c93a60541e3 Oct 30 22:27:10.768349 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.768317 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" event={"ID":"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489","Type":"ContainerStarted","Data":"5d162952a75deb24f9bcdaad4310ae2de478d53f082d331ec8f43c93a60541e3"} Oct 30 22:27:10.769966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.769943 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" event={"ID":"2056f2f9-5732-4c34-a715-5b7923c2cc93","Type":"ContainerStarted","Data":"3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354"} Oct 30 22:27:10.770058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.769988 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" event={"ID":"2056f2f9-5732-4c34-a715-5b7923c2cc93","Type":"ContainerStarted","Data":"6c3db7db42ba1bb73b01474c5f25046e324c73657aa79c18aef26ed326007e74"} Oct 30 22:27:10.770158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.770141 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:10.881539 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:10.881485 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" podStartSLOduration=1.8814701139999999 podStartE2EDuration="1.881470114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:10.877778419 +0000 UTC m=+1522.135120130" watchObservedRunningTime="2026-10-30 22:27:10.881470114 +0000 UTC m=+1522.138811823" Oct 30 22:27:11.779037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:11.778998 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" event={"ID":"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489","Type":"ContainerStarted","Data":"4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9"} Oct 30 22:27:11.779524 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:11.779069 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:11.889683 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:11.889579 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" podStartSLOduration=2.88955952 podStartE2EDuration="2.88955952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:11.887301932 +0000 UTC m=+1523.144643653" watchObservedRunningTime="2026-10-30 22:27:11.88955952 +0000 UTC m=+1523.146901230" Oct 30 22:27:21.782943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:21.782868 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:27:22.789474 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:22.789441 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:27:23.068406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:23.068318 5226 patch_prober.go:29] interesting pod/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:27:23.068406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:23.068386 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:27:24.412400 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:27:24.412364 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:27:25.910735 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:25.910704 5226 generic.go:356] "Generic (PLEG): container finished" podID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerID="9871b32e1b17df8c640fe7d3a4b22d307870827b8987ce51a5780ada8f5deaa8" exitCode=0 Oct 30 22:27:25.911192 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:25.910775 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" event={"ID":"014a257a-9a73-4c06-bb86-b00e10c032e5","Type":"ContainerDied","Data":"9871b32e1b17df8c640fe7d3a4b22d307870827b8987ce51a5780ada8f5deaa8"} Oct 30 22:27:25.911192 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:25.911147 5226 scope.go:122] "RemoveContainer" containerID="9871b32e1b17df8c640fe7d3a4b22d307870827b8987ce51a5780ada8f5deaa8" Oct 30 22:27:25.912756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:25.912725 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:25.912877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:25.912811 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:25.912877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:25.912859 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:27:26.919488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:26.919450 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" event={"ID":"014a257a-9a73-4c06-bb86-b00e10c032e5","Type":"ContainerStarted","Data":"de615a849e904f1c992a811e31c626dd9677cff5826e9da6f728a5357c30e614"} Oct 30 22:27:27.298113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:27.298079 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:27:27.298309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:27.298225 5226 patch_prober.go:29] interesting pod/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:27:27.298309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:27.298275 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:27:28.298544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:28.298504 5226 patch_prober.go:29] interesting pod/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:27:28.298930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:28.298567 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:27:29.297919 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:29.297843 5226 patch_prober.go:29] interesting pod/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:27:29.297919 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:29.297894 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:27:30.298489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:30.298455 5226 patch_prober.go:29] interesting pod/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:27:30.298888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:30.298503 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:27:31.298181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:31.298142 5226 patch_prober.go:29] interesting pod/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 22:27:31.298390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:31.298199 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 22:27:32.302753 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:32.302723 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:27:32.303235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:32.303051 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:27:32.303827 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:32.303812 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:27:48.530720 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:48.530683 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:27:54.412378 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:27:54.412336 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:27:57.818675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:57.818639 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq"] Oct 30 22:27:57.918871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:27:57.918837 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk"] Oct 30 22:28:02.819851 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:02.819790 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" podUID="0a283f99-0a43-4723-9826-481518aae650" containerName="payload-processing" containerID="cri-o://152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e" gracePeriod=25 Oct 30 22:28:02.920173 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:02.920119 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" podUID="4059d749-968e-4e37-a519-b166cb7418b1" containerName="payload-pre-processing" containerID="cri-o://3dc374d52e6dbea05db1213433d6c8f381a818396c6aec537db4a6e6aacec2c1" gracePeriod=25 Oct 30 22:28:03.123447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.123417 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:28:03.203760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.203726 5226 generic.go:356] "Generic (PLEG): container finished" podID="4059d749-968e-4e37-a519-b166cb7418b1" containerID="3dc374d52e6dbea05db1213433d6c8f381a818396c6aec537db4a6e6aacec2c1" exitCode=0 Oct 30 22:28:03.203961 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.203815 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" event={"ID":"4059d749-968e-4e37-a519-b166cb7418b1","Type":"ContainerDied","Data":"3dc374d52e6dbea05db1213433d6c8f381a818396c6aec537db4a6e6aacec2c1"} Oct 30 22:28:03.205363 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.205336 5226 generic.go:356] "Generic (PLEG): container finished" podID="0a283f99-0a43-4723-9826-481518aae650" containerID="152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e" exitCode=0 Oct 30 22:28:03.205504 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.205382 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" Oct 30 22:28:03.205504 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.205373 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" event={"ID":"0a283f99-0a43-4723-9826-481518aae650","Type":"ContainerDied","Data":"152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e"} Oct 30 22:28:03.205632 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.205531 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq" event={"ID":"0a283f99-0a43-4723-9826-481518aae650","Type":"ContainerDied","Data":"1b171f90d8b9f7bba7238a7b5121b2c8e893eadf56e7c63f036ab9221298980b"} Oct 30 22:28:03.205632 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.205568 5226 scope.go:122] "RemoveContainer" containerID="152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e" Oct 30 22:28:03.216313 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.216289 5226 scope.go:122] "RemoveContainer" containerID="152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e" Oct 30 22:28:03.216662 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:28:03.216636 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e\": container with ID starting with 152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e not found: ID does not exist" containerID="152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e" Oct 30 22:28:03.216761 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.216674 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e"} err="failed to get container status \"152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e\": rpc error: code = NotFound desc = could not find container \"152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e\": container with ID starting with 152b74b7af593bf9abfbdee0345c662d51307b5ef576ae95384bdee8ce80b46e not found: ID does not exist" Oct 30 22:28:03.240877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.240851 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:28:03.280518 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.280487 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a283f99-0a43-4723-9826-481518aae650-kube-api-access-t5sx7\" (UniqueName: \"kubernetes.io/projected/0a283f99-0a43-4723-9826-481518aae650-kube-api-access-t5sx7\") pod \"0a283f99-0a43-4723-9826-481518aae650\" (UID: \"0a283f99-0a43-4723-9826-481518aae650\") " Oct 30 22:28:03.280791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.280770 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a283f99-0a43-4723-9826-481518aae650-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a283f99-0a43-4723-9826-481518aae650-plugins-config-volume\") pod \"0a283f99-0a43-4723-9826-481518aae650\" (UID: \"0a283f99-0a43-4723-9826-481518aae650\") " Oct 30 22:28:03.281066 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.281044 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a283f99-0a43-4723-9826-481518aae650-plugins-config-volume" pod "0a283f99-0a43-4723-9826-481518aae650" (UID: "0a283f99-0a43-4723-9826-481518aae650"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:03.281459 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.281439 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a283f99-0a43-4723-9826-481518aae650-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:03.282828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.282810 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a283f99-0a43-4723-9826-481518aae650-kube-api-access-t5sx7" pod "0a283f99-0a43-4723-9826-481518aae650" (UID: "0a283f99-0a43-4723-9826-481518aae650"). InnerVolumeSpecName "kube-api-access-t5sx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:03.382493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.382416 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4059d749-968e-4e37-a519-b166cb7418b1-kube-api-access-dp7tl\" (UniqueName: \"kubernetes.io/projected/4059d749-968e-4e37-a519-b166cb7418b1-kube-api-access-dp7tl\") pod \"4059d749-968e-4e37-a519-b166cb7418b1\" (UID: \"4059d749-968e-4e37-a519-b166cb7418b1\") " Oct 30 22:28:03.382649 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.382634 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4059d749-968e-4e37-a519-b166cb7418b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4059d749-968e-4e37-a519-b166cb7418b1-plugins-config-volume\") pod \"4059d749-968e-4e37-a519-b166cb7418b1\" (UID: \"4059d749-968e-4e37-a519-b166cb7418b1\") " Oct 30 22:28:03.382981 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.382956 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4059d749-968e-4e37-a519-b166cb7418b1-plugins-config-volume" pod "4059d749-968e-4e37-a519-b166cb7418b1" (UID: "4059d749-968e-4e37-a519-b166cb7418b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:03.383346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.383327 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5sx7\" (UniqueName: \"kubernetes.io/projected/0a283f99-0a43-4723-9826-481518aae650-kube-api-access-t5sx7\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:03.383422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.383352 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4059d749-968e-4e37-a519-b166cb7418b1-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:03.384472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.384453 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4059d749-968e-4e37-a519-b166cb7418b1-kube-api-access-dp7tl" pod "4059d749-968e-4e37-a519-b166cb7418b1" (UID: "4059d749-968e-4e37-a519-b166cb7418b1"). InnerVolumeSpecName "kube-api-access-dp7tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:03.485284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.485237 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp7tl\" (UniqueName: \"kubernetes.io/projected/4059d749-968e-4e37-a519-b166cb7418b1-kube-api-access-dp7tl\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:03.621397 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.621364 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq"] Oct 30 22:28:03.626592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:03.626564 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cc8c7d75-754bc6fc9c-hwpsq"] Oct 30 22:28:04.214199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:04.214170 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" Oct 30 22:28:04.214703 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:04.214214 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk" event={"ID":"4059d749-968e-4e37-a519-b166cb7418b1","Type":"ContainerDied","Data":"446c099ad80f1da3d9742b0e83dcbe3d8e691e50300ef0b69717e10b64e5c2a5"} Oct 30 22:28:04.214703 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:04.214283 5226 scope.go:122] "RemoveContainer" containerID="3dc374d52e6dbea05db1213433d6c8f381a818396c6aec537db4a6e6aacec2c1" Oct 30 22:28:04.370181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:04.370141 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk"] Oct 30 22:28:04.415937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:04.415906 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cc8c7d75-59c6df8687-6ntlk"] Oct 30 22:28:04.907059 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:04.907028 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a283f99-0a43-4723-9826-481518aae650" path="/var/lib/kubelet/pods/0a283f99-0a43-4723-9826-481518aae650/volumes" Oct 30 22:28:04.907537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:04.907521 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4059d749-968e-4e37-a519-b166cb7418b1" path="/var/lib/kubelet/pods/4059d749-968e-4e37-a519-b166cb7418b1/volumes" Oct 30 22:28:11.302135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.302097 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc"] Oct 30 22:28:11.302948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.302927 5226 cpu_manager.go:414] "containerMap: removing container" podUID="4059d749-968e-4e37-a519-b166cb7418b1" containerName="payload-pre-processing" Oct 30 22:28:11.303019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.302950 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4059d749-968e-4e37-a519-b166cb7418b1" containerName="payload-pre-processing" Oct 30 22:28:11.303019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.302989 5226 cpu_manager.go:414] "containerMap: removing container" podUID="0a283f99-0a43-4723-9826-481518aae650" containerName="payload-processing" Oct 30 22:28:11.303019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.302994 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a283f99-0a43-4723-9826-481518aae650" containerName="payload-processing" Oct 30 22:28:11.303129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.303096 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="4059d749-968e-4e37-a519-b166cb7418b1" containerName="payload-pre-processing" Oct 30 22:28:11.303129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.303111 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a283f99-0a43-4723-9826-481518aae650" containerName="payload-processing" Oct 30 22:28:11.306848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.306825 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.364602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.364569 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-61346ee8-serving-cert\"" Oct 30 22:28:11.407762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.407713 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc"] Oct 30 22:28:11.427513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.427464 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.427710 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.427666 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f12682-fb05-4d24-867f-1866d54d928a-openshift-service-ca\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.427811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.427787 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwsl\" (UniqueName: \"kubernetes.io/projected/f7f12682-fb05-4d24-867f-1866d54d928a-kube-api-access-rrwsl\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.427937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.427920 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-metrics-certs\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.529226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.529183 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-metrics-certs\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.529422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.529329 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.529422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.529404 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f12682-fb05-4d24-867f-1866d54d928a-openshift-service-ca\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.529553 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:28:11.529471 5226 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-serving-cert: secret "maas-api-e2e-mt-61346ee8-serving-cert" not found Oct 30 22:28:11.529553 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.529475 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwsl\" (UniqueName: \"kubernetes.io/projected/f7f12682-fb05-4d24-867f-1866d54d928a-kube-api-access-rrwsl\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.529553 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:28:11.529536 5226 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls podName:f7f12682-fb05-4d24-867f-1866d54d928a nodeName:}" failed. No retries permitted until 2026-10-30 22:28:12.029516836 +0000 UTC m=+1583.286858533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls") pod "maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" (UID: "f7f12682-fb05-4d24-867f-1866d54d928a") : secret "maas-api-e2e-mt-61346ee8-serving-cert" not found Oct 30 22:28:11.530239 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.530191 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f12682-fb05-4d24-867f-1866d54d928a-openshift-service-ca\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.531578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.531518 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-metrics-certs\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:11.588841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:11.588765 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwsl\" (UniqueName: \"kubernetes.io/projected/f7f12682-fb05-4d24-867f-1866d54d928a-kube-api-access-rrwsl\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:12.041587 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:12.041549 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:12.044615 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:12.044577 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls\") pod \"maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:12.239532 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:12.239497 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:12.632031 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:12.631998 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc"] Oct 30 22:28:12.632603 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:28:12.632556 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f12682_fb05_4d24_867f_1866d54d928a.slice/crio-69b45d7ca4c4c8fca62bcd54d26c4d4309c2b607552909190db9f20a741f5ede WatchSource:0}: Error finding container 69b45d7ca4c4c8fca62bcd54d26c4d4309c2b607552909190db9f20a741f5ede: Status 404 returned error can't find the container with id 69b45d7ca4c4c8fca62bcd54d26c4d4309c2b607552909190db9f20a741f5ede Oct 30 22:28:13.353880 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:13.353833 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" event={"ID":"f7f12682-fb05-4d24-867f-1866d54d928a","Type":"ContainerStarted","Data":"69b45d7ca4c4c8fca62bcd54d26c4d4309c2b607552909190db9f20a741f5ede"} Oct 30 22:28:14.362978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:14.362945 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" event={"ID":"f7f12682-fb05-4d24-867f-1866d54d928a","Type":"ContainerStarted","Data":"286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6"} Oct 30 22:28:14.363417 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:14.363083 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:14.479755 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:14.479695 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" podStartSLOduration=2.584193986 podStartE2EDuration="3.479679502s" totalImagesPullingTime="895.485516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:14.47624141 +0000 UTC m=+1585.733583121" watchObservedRunningTime="2026-10-30 22:28:14.479679502 +0000 UTC m=+1585.737021211" Oct 30 22:28:20.376663 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:20.376633 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:28:24.411598 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:28:24.411559 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:28:48.432470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:48.432434 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc"] Oct 30 22:28:48.433433 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:48.433384 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" containerID="cri-o://de615a849e904f1c992a811e31c626dd9677cff5826e9da6f728a5357c30e614" gracePeriod=30 Oct 30 22:28:53.672013 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.671977 5226 generic.go:356] "Generic (PLEG): container finished" podID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerID="de615a849e904f1c992a811e31c626dd9677cff5826e9da6f728a5357c30e614" exitCode=0 Oct 30 22:28:53.672624 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.672047 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" event={"ID":"014a257a-9a73-4c06-bb86-b00e10c032e5","Type":"ContainerDied","Data":"de615a849e904f1c992a811e31c626dd9677cff5826e9da6f728a5357c30e614"} Oct 30 22:28:53.672624 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.672123 5226 scope.go:122] "RemoveContainer" containerID="9871b32e1b17df8c640fe7d3a4b22d307870827b8987ce51a5780ada8f5deaa8" Oct 30 22:28:53.727005 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.726945 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:28:53.869588 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869552 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-credential-socket\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.869775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869616 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-envoy\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.869775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869635 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-kube-api-access-tzp8v\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-kube-api-access-tzp8v\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.869775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869696 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-data\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.869775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869770 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-socket\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.869966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869799 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-certs\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.869966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869854 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-credential-socket" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:53.869966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869866 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-podinfo\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.869966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.869943 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014a257a-9a73-4c06-bb86-b00e10c032e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a257a-9a73-4c06-bb86-b00e10c032e5-istiod-ca-cert\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.870182 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.870149 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-socket" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:53.870270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.870185 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-token\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-token\") pod \"014a257a-9a73-4c06-bb86-b00e10c032e5\" (UID: \"014a257a-9a73-4c06-bb86-b00e10c032e5\") " Oct 30 22:28:53.870331 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.870259 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-certs" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:53.870331 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.870273 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-data" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:53.870635 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.870604 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014a257a-9a73-4c06-bb86-b00e10c032e5-istiod-ca-cert" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:53.871057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.871021 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.871057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.871051 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.871171 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.871067 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.871171 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.871083 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.871171 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.871098 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014a257a-9a73-4c06-bb86-b00e10c032e5-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.872121 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.872091 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-kube-api-access-tzp8v" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "kube-api-access-tzp8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:53.872280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.872262 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-token" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:53.872395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.872377 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-envoy" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:53.872450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.872430 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-podinfo" pod "014a257a-9a73-4c06-bb86-b00e10c032e5" (UID: "014a257a-9a73-4c06-bb86-b00e10c032e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:53.976292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.973867 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.976292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.973912 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.976292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.973932 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014a257a-9a73-4c06-bb86-b00e10c032e5-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:53.976292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:53.973948 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzp8v\" (UniqueName: \"kubernetes.io/projected/014a257a-9a73-4c06-bb86-b00e10c032e5-kube-api-access-tzp8v\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:54.440672 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:28:54.440633 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:28:54.680531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:54.680496 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" event={"ID":"014a257a-9a73-4c06-bb86-b00e10c032e5","Type":"ContainerDied","Data":"d92e38725cad6cee5ecaec4018d4889400bcd21d692c3b5fc22a0710c3b26a7f"} Oct 30 22:28:54.681025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:54.680561 5226 scope.go:122] "RemoveContainer" containerID="de615a849e904f1c992a811e31c626dd9677cff5826e9da6f728a5357c30e614" Oct 30 22:28:54.681025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:54.680514 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc" Oct 30 22:28:54.854026 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:54.849952 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc"] Oct 30 22:28:54.970181 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:54.970100 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cc8c7d75-gw-openshift-default-c77c885f7-5sjqc"] Oct 30 22:28:55.171919 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:55.171872 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:28:56.906548 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:28:56.906514 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" path="/var/lib/kubelet/pods/014a257a-9a73-4c06-bb86-b00e10c032e5/volumes" Oct 30 22:29:07.924227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:07.923964 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv"] Oct 30 22:29:07.928307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:07.925052 5226 cpu_manager.go:414] "containerMap: removing container" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" Oct 30 22:29:07.928307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:07.925077 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" Oct 30 22:29:07.928307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:07.925266 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" Oct 30 22:29:07.928307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:07.925284 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" Oct 30 22:29:07.931270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:07.931238 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:07.987729 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:07.987694 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv"] Oct 30 22:29:08.031680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.031629 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-73735c-dockercfg-lsnjs\"" Oct 30 22:29:08.032058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.032018 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-73735c\"" Oct 30 22:29:08.065844 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.065801 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a642-65b5-4f84-95aa-2cc3fe901416-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv\" (UID: \"1137a642-65b5-4f84-95aa-2cc3fe901416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:08.070068 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.070033 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bcst\" (UniqueName: \"kubernetes.io/projected/1137a642-65b5-4f84-95aa-2cc3fe901416-kube-api-access-6bcst\") pod \"payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv\" (UID: \"1137a642-65b5-4f84-95aa-2cc3fe901416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:08.225287 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.210601 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a642-65b5-4f84-95aa-2cc3fe901416-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv\" (UID: \"1137a642-65b5-4f84-95aa-2cc3fe901416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:08.225287 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.210721 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bcst\" (UniqueName: \"kubernetes.io/projected/1137a642-65b5-4f84-95aa-2cc3fe901416-kube-api-access-6bcst\") pod \"payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv\" (UID: \"1137a642-65b5-4f84-95aa-2cc3fe901416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:08.225287 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.212578 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a642-65b5-4f84-95aa-2cc3fe901416-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv\" (UID: \"1137a642-65b5-4f84-95aa-2cc3fe901416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:08.312338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.312300 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bcst\" (UniqueName: \"kubernetes.io/projected/1137a642-65b5-4f84-95aa-2cc3fe901416-kube-api-access-6bcst\") pod \"payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv\" (UID: \"1137a642-65b5-4f84-95aa-2cc3fe901416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:08.546449 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.546366 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:08.936220 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:08.936176 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv"] Oct 30 22:29:08.940790 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:29:08.940761 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1137a642_65b5_4f84_95aa_2cc3fe901416.slice/crio-73630eb7d692794294b46c4a61adec28c2327306e8508c1692acba0401a0593e WatchSource:0}: Error finding container 73630eb7d692794294b46c4a61adec28c2327306e8508c1692acba0401a0593e: Status 404 returned error can't find the container with id 73630eb7d692794294b46c4a61adec28c2327306e8508c1692acba0401a0593e Oct 30 22:29:09.869887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:09.869840 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" event={"ID":"1137a642-65b5-4f84-95aa-2cc3fe901416","Type":"ContainerStarted","Data":"5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d"} Oct 30 22:29:09.870164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:09.870139 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" event={"ID":"1137a642-65b5-4f84-95aa-2cc3fe901416","Type":"ContainerStarted","Data":"73630eb7d692794294b46c4a61adec28c2327306e8508c1692acba0401a0593e"} Oct 30 22:29:09.870300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:09.870181 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:09.983565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:09.983513 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" podStartSLOduration=2.983498301 podStartE2EDuration="2.983498301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:09.979162581 +0000 UTC m=+1641.236504292" watchObservedRunningTime="2026-10-30 22:29:09.983498301 +0000 UTC m=+1641.240840012" Oct 30 22:29:20.879845 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:29:20.879814 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:29:24.416190 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:29:24.416146 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:29:54.412410 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:29:54.412369 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:30:00.205893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.205853 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx"] Oct 30 22:30:00.207016 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.206993 5226 cpu_manager.go:414] "containerMap: removing container" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" Oct 30 22:30:00.207126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.207020 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014a257a-9a73-4c06-bb86-b00e10c032e5" containerName="istio-proxy" Oct 30 22:30:00.212765 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.212743 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf"] Oct 30 22:30:00.212963 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.212853 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:00.217453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.217430 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:00.217692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.217669 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s"] Oct 30 22:30:00.222344 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.222029 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:00.274567 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.272294 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw"] Oct 30 22:30:00.276765 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.276736 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb"] Oct 30 22:30:00.276923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.276867 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:00.281065 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.281039 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:00.291401 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.291366 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx"] Oct 30 22:30:00.304381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.304351 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf"] Oct 30 22:30:00.314545 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.314521 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s"] Oct 30 22:30:00.318116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.318093 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw"] Oct 30 22:30:00.324004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.323977 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76e61e-f47d-45b7-b679-099624d85bd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw\" (UID: \"ea76e61e-f47d-45b7-b679-099624d85bd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:00.324137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324098 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb778\" (UniqueName: \"kubernetes.io/projected/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-kube-api-access-vb778\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx\" (UID: \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:00.324239 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324139 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srj5\" (UniqueName: \"kubernetes.io/projected/1434b45d-095f-4996-9249-b3a66a296d4d-kube-api-access-2srj5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb\" (UID: \"1434b45d-095f-4996-9249-b3a66a296d4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:00.324311 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324260 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1434b45d-095f-4996-9249-b3a66a296d4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb\" (UID: \"1434b45d-095f-4996-9249-b3a66a296d4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:00.324311 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324291 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshqf\" (UniqueName: \"kubernetes.io/projected/9bb22783-c6ac-484c-a645-9b075a33cf60-kube-api-access-kshqf\") pod \"maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf\" (UID: \"9bb22783-c6ac-484c-a645-9b075a33cf60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:00.324454 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324436 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx\" (UID: \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:00.324513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324496 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb22783-c6ac-484c-a645-9b075a33cf60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf\" (UID: \"9bb22783-c6ac-484c-a645-9b075a33cf60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:00.324579 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324532 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6jp\" (UniqueName: \"kubernetes.io/projected/57422ef2-4bae-4536-a271-52aefac80ed7-kube-api-access-bq6jp\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s\" (UID: \"57422ef2-4bae-4536-a271-52aefac80ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:00.324634 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324574 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57422ef2-4bae-4536-a271-52aefac80ed7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s\" (UID: \"57422ef2-4bae-4536-a271-52aefac80ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:00.324691 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.324636 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdj6t\" (UniqueName: \"kubernetes.io/projected/ea76e61e-f47d-45b7-b679-099624d85bd1-kube-api-access-pdj6t\") pod \"maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw\" (UID: \"ea76e61e-f47d-45b7-b679-099624d85bd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:00.327798 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.327774 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb"] Oct 30 22:30:00.334609 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.334587 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\"" Oct 30 22:30:00.426247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426186 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdj6t\" (UniqueName: \"kubernetes.io/projected/ea76e61e-f47d-45b7-b679-099624d85bd1-kube-api-access-pdj6t\") pod \"maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw\" (UID: \"ea76e61e-f47d-45b7-b679-099624d85bd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:00.426441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426284 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76e61e-f47d-45b7-b679-099624d85bd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw\" (UID: \"ea76e61e-f47d-45b7-b679-099624d85bd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:00.426441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426357 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb778\" (UniqueName: \"kubernetes.io/projected/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-kube-api-access-vb778\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx\" (UID: \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:00.426441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426386 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srj5\" (UniqueName: \"kubernetes.io/projected/1434b45d-095f-4996-9249-b3a66a296d4d-kube-api-access-2srj5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb\" (UID: \"1434b45d-095f-4996-9249-b3a66a296d4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:00.426441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426427 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1434b45d-095f-4996-9249-b3a66a296d4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb\" (UID: \"1434b45d-095f-4996-9249-b3a66a296d4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:00.426441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426442 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshqf\" (UniqueName: \"kubernetes.io/projected/9bb22783-c6ac-484c-a645-9b075a33cf60-kube-api-access-kshqf\") pod \"maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf\" (UID: \"9bb22783-c6ac-484c-a645-9b075a33cf60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:00.426820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426605 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx\" (UID: \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:00.426820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426668 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb22783-c6ac-484c-a645-9b075a33cf60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf\" (UID: \"9bb22783-c6ac-484c-a645-9b075a33cf60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:00.426820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426708 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6jp\" (UniqueName: \"kubernetes.io/projected/57422ef2-4bae-4536-a271-52aefac80ed7-kube-api-access-bq6jp\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s\" (UID: \"57422ef2-4bae-4536-a271-52aefac80ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:00.426820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.426742 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57422ef2-4bae-4536-a271-52aefac80ed7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s\" (UID: \"57422ef2-4bae-4536-a271-52aefac80ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:00.427168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.427142 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76e61e-f47d-45b7-b679-099624d85bd1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw\" (UID: \"ea76e61e-f47d-45b7-b679-099624d85bd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:00.427283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.427192 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1434b45d-095f-4996-9249-b3a66a296d4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb\" (UID: \"1434b45d-095f-4996-9249-b3a66a296d4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:00.427450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.427429 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx\" (UID: \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:00.427491 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.427440 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57422ef2-4bae-4536-a271-52aefac80ed7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s\" (UID: \"57422ef2-4bae-4536-a271-52aefac80ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:00.427876 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.427860 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb22783-c6ac-484c-a645-9b075a33cf60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf\" (UID: \"9bb22783-c6ac-484c-a645-9b075a33cf60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:00.484126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.484056 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshqf\" (UniqueName: \"kubernetes.io/projected/9bb22783-c6ac-484c-a645-9b075a33cf60-kube-api-access-kshqf\") pod \"maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf\" (UID: \"9bb22783-c6ac-484c-a645-9b075a33cf60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:00.484126 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.484058 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb778\" (UniqueName: \"kubernetes.io/projected/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-kube-api-access-vb778\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx\" (UID: \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:00.484341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.484234 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdj6t\" (UniqueName: \"kubernetes.io/projected/ea76e61e-f47d-45b7-b679-099624d85bd1-kube-api-access-pdj6t\") pod \"maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw\" (UID: \"ea76e61e-f47d-45b7-b679-099624d85bd1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:00.484341 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.484300 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6jp\" (UniqueName: \"kubernetes.io/projected/57422ef2-4bae-4536-a271-52aefac80ed7-kube-api-access-bq6jp\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s\" (UID: \"57422ef2-4bae-4536-a271-52aefac80ed7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:00.484415 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.484354 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srj5\" (UniqueName: \"kubernetes.io/projected/1434b45d-095f-4996-9249-b3a66a296d4d-kube-api-access-2srj5\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb\" (UID: \"1434b45d-095f-4996-9249-b3a66a296d4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:00.530365 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.530334 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:00.538175 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.538152 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:00.547132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.547109 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:00.623364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.622389 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:00.623364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.622781 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:00.939859 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.939826 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx"] Oct 30 22:30:00.944518 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:30:00.944491 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe7bbd0_1a6b_4524_bee5_7ee94670041e.slice/crio-3a19e42679ccafcecb841b92d6f08a7b17071ba370c48fcbebde64d3945ee108 WatchSource:0}: Error finding container 3a19e42679ccafcecb841b92d6f08a7b17071ba370c48fcbebde64d3945ee108: Status 404 returned error can't find the container with id 3a19e42679ccafcecb841b92d6f08a7b17071ba370c48fcbebde64d3945ee108 Oct 30 22:30:00.990957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.990933 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s"] Oct 30 22:30:00.994340 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:30:00.994316 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57422ef2_4bae_4536_a271_52aefac80ed7.slice/crio-0a77d9922be36bc5c6a52ff08e851df94df7c3398cdeddd127723f58f2919541 WatchSource:0}: Error finding container 0a77d9922be36bc5c6a52ff08e851df94df7c3398cdeddd127723f58f2919541: Status 404 returned error can't find the container with id 0a77d9922be36bc5c6a52ff08e851df94df7c3398cdeddd127723f58f2919541 Oct 30 22:30:00.995099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:00.995077 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf"] Oct 30 22:30:00.996370 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:30:00.996344 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb22783_c6ac_484c_a645_9b075a33cf60.slice/crio-79e5b1af1fa9619e08bf5989c6b49f393ac3cccf6aa47787329a9eafab529748 WatchSource:0}: Error finding container 79e5b1af1fa9619e08bf5989c6b49f393ac3cccf6aa47787329a9eafab529748: Status 404 returned error can't find the container with id 79e5b1af1fa9619e08bf5989c6b49f393ac3cccf6aa47787329a9eafab529748 Oct 30 22:30:01.042401 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:01.042375 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw"] Oct 30 22:30:01.046650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:01.046625 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb"] Oct 30 22:30:01.048256 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:30:01.048230 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea76e61e_f47d_45b7_b679_099624d85bd1.slice/crio-4772e660db3d4e8b66cae23b0b1ef82e6c5cb3439a84fd417173e085f36b2d7b WatchSource:0}: Error finding container 4772e660db3d4e8b66cae23b0b1ef82e6c5cb3439a84fd417173e085f36b2d7b: Status 404 returned error can't find the container with id 4772e660db3d4e8b66cae23b0b1ef82e6c5cb3439a84fd417173e085f36b2d7b Oct 30 22:30:01.049987 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:30:01.049959 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1434b45d_095f_4996_9249_b3a66a296d4d.slice/crio-13bb8a15125aa6386604f565a1ec7ce46abbfe0d99bea65d247b0e88c481aa7e WatchSource:0}: Error finding container 13bb8a15125aa6386604f565a1ec7ce46abbfe0d99bea65d247b0e88c481aa7e: Status 404 returned error can't find the container with id 13bb8a15125aa6386604f565a1ec7ce46abbfe0d99bea65d247b0e88c481aa7e Oct 30 22:30:01.361962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:01.361923 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" event={"ID":"57422ef2-4bae-4536-a271-52aefac80ed7","Type":"ContainerStarted","Data":"0a77d9922be36bc5c6a52ff08e851df94df7c3398cdeddd127723f58f2919541"} Oct 30 22:30:01.363792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:01.363766 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" event={"ID":"5fe7bbd0-1a6b-4524-bee5-7ee94670041e","Type":"ContainerStarted","Data":"3a19e42679ccafcecb841b92d6f08a7b17071ba370c48fcbebde64d3945ee108"} Oct 30 22:30:01.365087 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:01.365064 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" event={"ID":"1434b45d-095f-4996-9249-b3a66a296d4d","Type":"ContainerStarted","Data":"13bb8a15125aa6386604f565a1ec7ce46abbfe0d99bea65d247b0e88c481aa7e"} Oct 30 22:30:01.366352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:01.366327 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" event={"ID":"9bb22783-c6ac-484c-a645-9b075a33cf60","Type":"ContainerStarted","Data":"79e5b1af1fa9619e08bf5989c6b49f393ac3cccf6aa47787329a9eafab529748"} Oct 30 22:30:01.367492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:01.367472 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" event={"ID":"ea76e61e-f47d-45b7-b679-099624d85bd1","Type":"ContainerStarted","Data":"4772e660db3d4e8b66cae23b0b1ef82e6c5cb3439a84fd417173e085f36b2d7b"} Oct 30 22:30:02.375855 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.375816 5226 generic.go:356] "Generic (PLEG): container finished" podID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerID="d79207d1ca9bfd37e3f4d7ddc778ac3ebfe3be72936a68e7569c29a2b77eda1a" exitCode=60 Oct 30 22:30:02.376317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.375905 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" event={"ID":"5fe7bbd0-1a6b-4524-bee5-7ee94670041e","Type":"ContainerDied","Data":"d79207d1ca9bfd37e3f4d7ddc778ac3ebfe3be72936a68e7569c29a2b77eda1a"} Oct 30 22:30:02.376317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.376259 5226 scope.go:122] "RemoveContainer" containerID="d79207d1ca9bfd37e3f4d7ddc778ac3ebfe3be72936a68e7569c29a2b77eda1a" Oct 30 22:30:02.377874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.377847 5226 generic.go:356] "Generic (PLEG): container finished" podID="1434b45d-095f-4996-9249-b3a66a296d4d" containerID="dbe5a5bb1f3fd866ad403e3bd52c96262ad00c2919b2f1ee91e54e5d70dcb7c2" exitCode=60 Oct 30 22:30:02.378008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.377895 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" event={"ID":"1434b45d-095f-4996-9249-b3a66a296d4d","Type":"ContainerDied","Data":"dbe5a5bb1f3fd866ad403e3bd52c96262ad00c2919b2f1ee91e54e5d70dcb7c2"} Oct 30 22:30:02.378293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.378266 5226 scope.go:122] "RemoveContainer" containerID="dbe5a5bb1f3fd866ad403e3bd52c96262ad00c2919b2f1ee91e54e5d70dcb7c2" Oct 30 22:30:02.379846 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.379813 5226 generic.go:356] "Generic (PLEG): container finished" podID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerID="606091f203a65717077e118e7d59f62cf6728c2a0e601d700ed595c770c1d52b" exitCode=60 Oct 30 22:30:02.380086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.379913 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" event={"ID":"9bb22783-c6ac-484c-a645-9b075a33cf60","Type":"ContainerDied","Data":"606091f203a65717077e118e7d59f62cf6728c2a0e601d700ed595c770c1d52b"} Oct 30 22:30:02.380289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.380270 5226 scope.go:122] "RemoveContainer" containerID="606091f203a65717077e118e7d59f62cf6728c2a0e601d700ed595c770c1d52b" Oct 30 22:30:02.381745 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.381723 5226 generic.go:356] "Generic (PLEG): container finished" podID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerID="7f531d5360c9966350bc37db256532865fa37ea1f115957e05b6bf80b4187447" exitCode=60 Oct 30 22:30:02.381890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.381812 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" event={"ID":"ea76e61e-f47d-45b7-b679-099624d85bd1","Type":"ContainerDied","Data":"7f531d5360c9966350bc37db256532865fa37ea1f115957e05b6bf80b4187447"} Oct 30 22:30:02.382092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.382075 5226 scope.go:122] "RemoveContainer" containerID="7f531d5360c9966350bc37db256532865fa37ea1f115957e05b6bf80b4187447" Oct 30 22:30:02.383502 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.383483 5226 generic.go:356] "Generic (PLEG): container finished" podID="57422ef2-4bae-4536-a271-52aefac80ed7" containerID="1d649789a6267a3560aea8b44f75f483854cf2e900cc437cf98a41cd9b7cf325" exitCode=60 Oct 30 22:30:02.383584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.383515 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" event={"ID":"57422ef2-4bae-4536-a271-52aefac80ed7","Type":"ContainerDied","Data":"1d649789a6267a3560aea8b44f75f483854cf2e900cc437cf98a41cd9b7cf325"} Oct 30 22:30:02.383812 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:02.383776 5226 scope.go:122] "RemoveContainer" containerID="1d649789a6267a3560aea8b44f75f483854cf2e900cc437cf98a41cd9b7cf325" Oct 30 22:30:03.392295 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.392188 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" event={"ID":"57422ef2-4bae-4536-a271-52aefac80ed7","Type":"ContainerStarted","Data":"67c1ca6e82aebd63d436559f61e1aff6fdb77fb3bdb842671fe598560185cd44"} Oct 30 22:30:03.394195 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.394160 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" event={"ID":"5fe7bbd0-1a6b-4524-bee5-7ee94670041e","Type":"ContainerStarted","Data":"61460d86ad6ca51c766e1e880f2967bae005c691493e959f5d91214e39c37eda"} Oct 30 22:30:03.395999 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.395980 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" event={"ID":"1434b45d-095f-4996-9249-b3a66a296d4d","Type":"ContainerStarted","Data":"eb7ab94572ac5f45ea4abcbd827be010e71ef0e1b93a9cf4b82abb911820ab93"} Oct 30 22:30:03.397887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.397865 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" event={"ID":"9bb22783-c6ac-484c-a645-9b075a33cf60","Type":"ContainerStarted","Data":"c7a29194649789404e5125b0230828b5e260994ad2071720422f857b8a45395a"} Oct 30 22:30:03.399663 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.399642 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" event={"ID":"ea76e61e-f47d-45b7-b679-099624d85bd1","Type":"ContainerStarted","Data":"68f37be0d82bcd8d8f50a08929f758472b655ffdec9de8fd7cf6e84087cbd1eb"} Oct 30 22:30:03.501150 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.501098 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" podStartSLOduration=3.501085348 podStartE2EDuration="3.501085348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.499220016 +0000 UTC m=+1694.756561717" watchObservedRunningTime="2026-10-30 22:30:03.501085348 +0000 UTC m=+1694.758427058" Oct 30 22:30:03.607965 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.607910 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" podStartSLOduration=3.607894171 podStartE2EDuration="3.607894171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.60672894 +0000 UTC m=+1694.864070651" watchObservedRunningTime="2026-10-30 22:30:03.607894171 +0000 UTC m=+1694.865235880" Oct 30 22:30:03.724813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.724761 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" podStartSLOduration=3.7247475789999998 podStartE2EDuration="3.724747579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.721513151 +0000 UTC m=+1694.978854862" watchObservedRunningTime="2026-10-30 22:30:03.724747579 +0000 UTC m=+1694.982089299" Oct 30 22:30:03.851461 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.851398 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" podStartSLOduration=3.851381334 podStartE2EDuration="3.851381334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.83522246 +0000 UTC m=+1695.092564163" watchObservedRunningTime="2026-10-30 22:30:03.851381334 +0000 UTC m=+1695.108723053" Oct 30 22:30:03.948730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:03.948671 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" podStartSLOduration=3.948655828 podStartE2EDuration="3.948655828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.945473884 +0000 UTC m=+1695.202815595" watchObservedRunningTime="2026-10-30 22:30:03.948655828 +0000 UTC m=+1695.205997546" Oct 30 22:30:04.408934 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.408841 5226 generic.go:356] "Generic (PLEG): container finished" podID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerID="61460d86ad6ca51c766e1e880f2967bae005c691493e959f5d91214e39c37eda" exitCode=60 Oct 30 22:30:04.409404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.408923 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" event={"ID":"5fe7bbd0-1a6b-4524-bee5-7ee94670041e","Type":"ContainerDied","Data":"61460d86ad6ca51c766e1e880f2967bae005c691493e959f5d91214e39c37eda"} Oct 30 22:30:04.409404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.408993 5226 scope.go:122] "RemoveContainer" containerID="d79207d1ca9bfd37e3f4d7ddc778ac3ebfe3be72936a68e7569c29a2b77eda1a" Oct 30 22:30:04.409404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.409328 5226 scope.go:122] "RemoveContainer" containerID="61460d86ad6ca51c766e1e880f2967bae005c691493e959f5d91214e39c37eda" Oct 30 22:30:04.409737 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:04.409710 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx_odh-ai-gateway-infra(5fe7bbd0-1a6b-4524-bee5-7ee94670041e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" Oct 30 22:30:04.411245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.411189 5226 generic.go:356] "Generic (PLEG): container finished" podID="1434b45d-095f-4996-9249-b3a66a296d4d" containerID="eb7ab94572ac5f45ea4abcbd827be010e71ef0e1b93a9cf4b82abb911820ab93" exitCode=60 Oct 30 22:30:04.411370 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.411260 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" event={"ID":"1434b45d-095f-4996-9249-b3a66a296d4d","Type":"ContainerDied","Data":"eb7ab94572ac5f45ea4abcbd827be010e71ef0e1b93a9cf4b82abb911820ab93"} Oct 30 22:30:04.411640 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.411621 5226 scope.go:122] "RemoveContainer" containerID="eb7ab94572ac5f45ea4abcbd827be010e71ef0e1b93a9cf4b82abb911820ab93" Oct 30 22:30:04.412044 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:04.412018 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb_odh-ai-gateway-infra(1434b45d-095f-4996-9249-b3a66a296d4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" Oct 30 22:30:04.413450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.413425 5226 generic.go:356] "Generic (PLEG): container finished" podID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerID="c7a29194649789404e5125b0230828b5e260994ad2071720422f857b8a45395a" exitCode=60 Oct 30 22:30:04.413538 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.413500 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" event={"ID":"9bb22783-c6ac-484c-a645-9b075a33cf60","Type":"ContainerDied","Data":"c7a29194649789404e5125b0230828b5e260994ad2071720422f857b8a45395a"} Oct 30 22:30:04.413762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.413747 5226 scope.go:122] "RemoveContainer" containerID="c7a29194649789404e5125b0230828b5e260994ad2071720422f857b8a45395a" Oct 30 22:30:04.414095 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:04.414077 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf_odh-ai-gateway-infra(9bb22783-c6ac-484c-a645-9b075a33cf60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" Oct 30 22:30:04.415763 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.415719 5226 generic.go:356] "Generic (PLEG): container finished" podID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerID="68f37be0d82bcd8d8f50a08929f758472b655ffdec9de8fd7cf6e84087cbd1eb" exitCode=60 Oct 30 22:30:04.415848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.415807 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" event={"ID":"ea76e61e-f47d-45b7-b679-099624d85bd1","Type":"ContainerDied","Data":"68f37be0d82bcd8d8f50a08929f758472b655ffdec9de8fd7cf6e84087cbd1eb"} Oct 30 22:30:04.416055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.416039 5226 scope.go:122] "RemoveContainer" containerID="68f37be0d82bcd8d8f50a08929f758472b655ffdec9de8fd7cf6e84087cbd1eb" Oct 30 22:30:04.416405 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:04.416383 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw_odh-ai-gateway-infra(ea76e61e-f47d-45b7-b679-099624d85bd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" Oct 30 22:30:04.418034 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.418013 5226 generic.go:356] "Generic (PLEG): container finished" podID="57422ef2-4bae-4536-a271-52aefac80ed7" containerID="67c1ca6e82aebd63d436559f61e1aff6fdb77fb3bdb842671fe598560185cd44" exitCode=60 Oct 30 22:30:04.418120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.418051 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" event={"ID":"57422ef2-4bae-4536-a271-52aefac80ed7","Type":"ContainerDied","Data":"67c1ca6e82aebd63d436559f61e1aff6fdb77fb3bdb842671fe598560185cd44"} Oct 30 22:30:04.418391 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.418374 5226 scope.go:122] "RemoveContainer" containerID="67c1ca6e82aebd63d436559f61e1aff6fdb77fb3bdb842671fe598560185cd44" Oct 30 22:30:04.418739 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:04.418716 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s_odh-ai-gateway-infra(57422ef2-4bae-4536-a271-52aefac80ed7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" Oct 30 22:30:04.423728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.423712 5226 scope.go:122] "RemoveContainer" containerID="dbe5a5bb1f3fd866ad403e3bd52c96262ad00c2919b2f1ee91e54e5d70dcb7c2" Oct 30 22:30:04.440459 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.440432 5226 scope.go:122] "RemoveContainer" containerID="606091f203a65717077e118e7d59f62cf6728c2a0e601d700ed595c770c1d52b" Oct 30 22:30:04.466170 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.466146 5226 scope.go:122] "RemoveContainer" containerID="7f531d5360c9966350bc37db256532865fa37ea1f115957e05b6bf80b4187447" Oct 30 22:30:04.482917 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:04.482893 5226 scope.go:122] "RemoveContainer" containerID="1d649789a6267a3560aea8b44f75f483854cf2e900cc437cf98a41cd9b7cf325" Oct 30 22:30:05.427417 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:05.427381 5226 scope.go:122] "RemoveContainer" containerID="eb7ab94572ac5f45ea4abcbd827be010e71ef0e1b93a9cf4b82abb911820ab93" Oct 30 22:30:05.427882 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:05.427735 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb_odh-ai-gateway-infra(1434b45d-095f-4996-9249-b3a66a296d4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" Oct 30 22:30:05.428983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:05.428959 5226 scope.go:122] "RemoveContainer" containerID="c7a29194649789404e5125b0230828b5e260994ad2071720422f857b8a45395a" Oct 30 22:30:05.429337 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:05.429315 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf_odh-ai-gateway-infra(9bb22783-c6ac-484c-a645-9b075a33cf60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" Oct 30 22:30:05.430932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:05.430911 5226 scope.go:122] "RemoveContainer" containerID="68f37be0d82bcd8d8f50a08929f758472b655ffdec9de8fd7cf6e84087cbd1eb" Oct 30 22:30:05.431260 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:05.431239 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw_odh-ai-gateway-infra(ea76e61e-f47d-45b7-b679-099624d85bd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" Oct 30 22:30:05.432660 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:05.432630 5226 scope.go:122] "RemoveContainer" containerID="67c1ca6e82aebd63d436559f61e1aff6fdb77fb3bdb842671fe598560185cd44" Oct 30 22:30:05.432913 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:05.432888 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s_odh-ai-gateway-infra(57422ef2-4bae-4536-a271-52aefac80ed7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" Oct 30 22:30:05.434184 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:05.434167 5226 scope.go:122] "RemoveContainer" containerID="61460d86ad6ca51c766e1e880f2967bae005c691493e959f5d91214e39c37eda" Oct 30 22:30:05.434439 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:05.434422 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx_odh-ai-gateway-infra(5fe7bbd0-1a6b-4524-bee5-7ee94670041e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" Oct 30 22:30:13.899809 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:13.899774 5226 scope.go:122] "RemoveContainer" containerID="67c1ca6e82aebd63d436559f61e1aff6fdb77fb3bdb842671fe598560185cd44" Oct 30 22:30:13.903020 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:13.902998 5226 scope.go:122] "RemoveContainer" containerID="68f37be0d82bcd8d8f50a08929f758472b655ffdec9de8fd7cf6e84087cbd1eb" Oct 30 22:30:14.509670 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:14.509518 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" event={"ID":"57422ef2-4bae-4536-a271-52aefac80ed7","Type":"ContainerStarted","Data":"4876589540a9c49573b33330919b32e897e69826483f6635fac8dd8d4c2eba38"} Oct 30 22:30:14.898254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:14.898141 5226 scope.go:122] "RemoveContainer" containerID="61460d86ad6ca51c766e1e880f2967bae005c691493e959f5d91214e39c37eda" Oct 30 22:30:15.519814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.519730 5226 generic.go:356] "Generic (PLEG): container finished" podID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerID="9ee8ee81aaecf95a7d710f0e99272800b189739cc3cd194afe780ddae03d940c" exitCode=60 Oct 30 22:30:15.520277 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.519803 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" event={"ID":"ea76e61e-f47d-45b7-b679-099624d85bd1","Type":"ContainerDied","Data":"9ee8ee81aaecf95a7d710f0e99272800b189739cc3cd194afe780ddae03d940c"} Oct 30 22:30:15.520277 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.519870 5226 scope.go:122] "RemoveContainer" containerID="68f37be0d82bcd8d8f50a08929f758472b655ffdec9de8fd7cf6e84087cbd1eb" Oct 30 22:30:15.520383 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.520345 5226 scope.go:122] "RemoveContainer" containerID="9ee8ee81aaecf95a7d710f0e99272800b189739cc3cd194afe780ddae03d940c" Oct 30 22:30:15.520879 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:15.520734 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw_odh-ai-gateway-infra(ea76e61e-f47d-45b7-b679-099624d85bd1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" Oct 30 22:30:15.521994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.521973 5226 generic.go:356] "Generic (PLEG): container finished" podID="57422ef2-4bae-4536-a271-52aefac80ed7" containerID="4876589540a9c49573b33330919b32e897e69826483f6635fac8dd8d4c2eba38" exitCode=60 Oct 30 22:30:15.522076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.522051 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" event={"ID":"57422ef2-4bae-4536-a271-52aefac80ed7","Type":"ContainerDied","Data":"4876589540a9c49573b33330919b32e897e69826483f6635fac8dd8d4c2eba38"} Oct 30 22:30:15.522493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.522477 5226 scope.go:122] "RemoveContainer" containerID="4876589540a9c49573b33330919b32e897e69826483f6635fac8dd8d4c2eba38" Oct 30 22:30:15.522818 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:15.522799 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s_odh-ai-gateway-infra(57422ef2-4bae-4536-a271-52aefac80ed7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" Oct 30 22:30:15.523819 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.523800 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" event={"ID":"5fe7bbd0-1a6b-4524-bee5-7ee94670041e","Type":"ContainerStarted","Data":"3156bb9e03988616e705d9e3f79e14013de26dc8523d7a9586b80f25b6777869"} Oct 30 22:30:15.533042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.533024 5226 scope.go:122] "RemoveContainer" containerID="67c1ca6e82aebd63d436559f61e1aff6fdb77fb3bdb842671fe598560185cd44" Oct 30 22:30:15.675042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:15.674998 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s"] Oct 30 22:30:16.534004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.533972 5226 generic.go:356] "Generic (PLEG): container finished" podID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerID="3156bb9e03988616e705d9e3f79e14013de26dc8523d7a9586b80f25b6777869" exitCode=60 Oct 30 22:30:16.534556 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.534041 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" event={"ID":"5fe7bbd0-1a6b-4524-bee5-7ee94670041e","Type":"ContainerDied","Data":"3156bb9e03988616e705d9e3f79e14013de26dc8523d7a9586b80f25b6777869"} Oct 30 22:30:16.534556 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.534102 5226 scope.go:122] "RemoveContainer" containerID="61460d86ad6ca51c766e1e880f2967bae005c691493e959f5d91214e39c37eda" Oct 30 22:30:16.534556 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.534553 5226 scope.go:122] "RemoveContainer" containerID="3156bb9e03988616e705d9e3f79e14013de26dc8523d7a9586b80f25b6777869" Oct 30 22:30:16.534979 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:16.534948 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx_odh-ai-gateway-infra(5fe7bbd0-1a6b-4524-bee5-7ee94670041e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" Oct 30 22:30:16.642394 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.642335 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw"] Oct 30 22:30:16.712573 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.712481 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:16.750725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.750690 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx"] Oct 30 22:30:16.815601 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.815577 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57422ef2-4bae-4536-a271-52aefac80ed7-kube-api-access-bq6jp\" (UniqueName: \"kubernetes.io/projected/57422ef2-4bae-4536-a271-52aefac80ed7-kube-api-access-bq6jp\") pod \"57422ef2-4bae-4536-a271-52aefac80ed7\" (UID: \"57422ef2-4bae-4536-a271-52aefac80ed7\") " Oct 30 22:30:16.815740 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.815670 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57422ef2-4bae-4536-a271-52aefac80ed7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57422ef2-4bae-4536-a271-52aefac80ed7-maas-api-service-ca\") pod \"57422ef2-4bae-4536-a271-52aefac80ed7\" (UID: \"57422ef2-4bae-4536-a271-52aefac80ed7\") " Oct 30 22:30:16.816063 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.816040 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57422ef2-4bae-4536-a271-52aefac80ed7-maas-api-service-ca" pod "57422ef2-4bae-4536-a271-52aefac80ed7" (UID: "57422ef2-4bae-4536-a271-52aefac80ed7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:16.816386 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.816371 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57422ef2-4bae-4536-a271-52aefac80ed7-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.817764 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.817744 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57422ef2-4bae-4536-a271-52aefac80ed7-kube-api-access-bq6jp" pod "57422ef2-4bae-4536-a271-52aefac80ed7" (UID: "57422ef2-4bae-4536-a271-52aefac80ed7"). InnerVolumeSpecName "kube-api-access-bq6jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:16.844497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.844470 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:16.898338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.898310 5226 scope.go:122] "RemoveContainer" containerID="eb7ab94572ac5f45ea4abcbd827be010e71ef0e1b93a9cf4b82abb911820ab93" Oct 30 22:30:16.916864 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.916844 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea76e61e-f47d-45b7-b679-099624d85bd1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76e61e-f47d-45b7-b679-099624d85bd1-maas-api-service-ca\") pod \"ea76e61e-f47d-45b7-b679-099624d85bd1\" (UID: \"ea76e61e-f47d-45b7-b679-099624d85bd1\") " Oct 30 22:30:16.917118 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.917097 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea76e61e-f47d-45b7-b679-099624d85bd1-kube-api-access-pdj6t\" (UniqueName: \"kubernetes.io/projected/ea76e61e-f47d-45b7-b679-099624d85bd1-kube-api-access-pdj6t\") pod \"ea76e61e-f47d-45b7-b679-099624d85bd1\" (UID: \"ea76e61e-f47d-45b7-b679-099624d85bd1\") " Oct 30 22:30:16.917279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.917250 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea76e61e-f47d-45b7-b679-099624d85bd1-maas-api-service-ca" pod "ea76e61e-f47d-45b7-b679-099624d85bd1" (UID: "ea76e61e-f47d-45b7-b679-099624d85bd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:16.917706 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.917683 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76e61e-f47d-45b7-b679-099624d85bd1-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.917812 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.917712 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6jp\" (UniqueName: \"kubernetes.io/projected/57422ef2-4bae-4536-a271-52aefac80ed7-kube-api-access-bq6jp\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.919162 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:16.919141 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea76e61e-f47d-45b7-b679-099624d85bd1-kube-api-access-pdj6t" pod "ea76e61e-f47d-45b7-b679-099624d85bd1" (UID: "ea76e61e-f47d-45b7-b679-099624d85bd1"). InnerVolumeSpecName "kube-api-access-pdj6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.020090 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.020056 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdj6t\" (UniqueName: \"kubernetes.io/projected/ea76e61e-f47d-45b7-b679-099624d85bd1-kube-api-access-pdj6t\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.544541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.544452 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" Oct 30 22:30:17.544541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.544494 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw" event={"ID":"ea76e61e-f47d-45b7-b679-099624d85bd1","Type":"ContainerDied","Data":"4772e660db3d4e8b66cae23b0b1ef82e6c5cb3439a84fd417173e085f36b2d7b"} Oct 30 22:30:17.545086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.544604 5226 scope.go:122] "RemoveContainer" containerID="9ee8ee81aaecf95a7d710f0e99272800b189739cc3cd194afe780ddae03d940c" Oct 30 22:30:17.545955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.545920 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" Oct 30 22:30:17.545955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.545924 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s" event={"ID":"57422ef2-4bae-4536-a271-52aefac80ed7","Type":"ContainerDied","Data":"0a77d9922be36bc5c6a52ff08e851df94df7c3398cdeddd127723f58f2919541"} Oct 30 22:30:17.550502 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.550472 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" event={"ID":"1434b45d-095f-4996-9249-b3a66a296d4d","Type":"ContainerStarted","Data":"56c8ff28ec4c99e1995019f26874c4e11814a6e5c26e50b883207194f94c58d0"} Oct 30 22:30:17.558160 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.558132 5226 scope.go:122] "RemoveContainer" containerID="4876589540a9c49573b33330919b32e897e69826483f6635fac8dd8d4c2eba38" Oct 30 22:30:17.731533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.731509 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:17.808593 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.808497 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s"] Oct 30 22:30:17.812450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.812419 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29889990-46n7s"] Oct 30 22:30:17.832478 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.832446 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-kube-api-access-vb778\" (UniqueName: \"kubernetes.io/projected/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-kube-api-access-vb778\") pod \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\" (UID: \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\") " Oct 30 22:30:17.832638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.832524 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-maas-api-service-ca\") pod \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\" (UID: \"5fe7bbd0-1a6b-4524-bee5-7ee94670041e\") " Oct 30 22:30:17.832870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.832849 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-maas-api-service-ca" pod "5fe7bbd0-1a6b-4524-bee5-7ee94670041e" (UID: "5fe7bbd0-1a6b-4524-bee5-7ee94670041e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:17.833280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.833261 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:17.834674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.834654 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-kube-api-access-vb778" pod "5fe7bbd0-1a6b-4524-bee5-7ee94670041e" (UID: "5fe7bbd0-1a6b-4524-bee5-7ee94670041e"). InnerVolumeSpecName "kube-api-access-vb778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:17.898339 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.898306 5226 scope.go:122] "RemoveContainer" containerID="c7a29194649789404e5125b0230828b5e260994ad2071720422f857b8a45395a" Oct 30 22:30:17.934750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:17.934722 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb778\" (UniqueName: \"kubernetes.io/projected/5fe7bbd0-1a6b-4524-bee5-7ee94670041e-kube-api-access-vb778\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.024899 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.024850 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw"] Oct 30 22:30:18.028797 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.028769 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-61346ee8-29889990-w2lfw"] Oct 30 22:30:18.559465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.559358 5226 generic.go:356] "Generic (PLEG): container finished" podID="1434b45d-095f-4996-9249-b3a66a296d4d" containerID="56c8ff28ec4c99e1995019f26874c4e11814a6e5c26e50b883207194f94c58d0" exitCode=60 Oct 30 22:30:18.559465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.559438 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" event={"ID":"1434b45d-095f-4996-9249-b3a66a296d4d","Type":"ContainerDied","Data":"56c8ff28ec4c99e1995019f26874c4e11814a6e5c26e50b883207194f94c58d0"} Oct 30 22:30:18.560453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.559519 5226 scope.go:122] "RemoveContainer" containerID="eb7ab94572ac5f45ea4abcbd827be010e71ef0e1b93a9cf4b82abb911820ab93" Oct 30 22:30:18.560453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.560194 5226 scope.go:122] "RemoveContainer" containerID="56c8ff28ec4c99e1995019f26874c4e11814a6e5c26e50b883207194f94c58d0" Oct 30 22:30:18.560638 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:18.560617 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb_odh-ai-gateway-infra(1434b45d-095f-4996-9249-b3a66a296d4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" Oct 30 22:30:18.562375 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.562048 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" event={"ID":"9bb22783-c6ac-484c-a645-9b075a33cf60","Type":"ContainerStarted","Data":"add0f8a466ee4d822bb5ebdca1c8cdf9bcdfd385f96b4d6587165b0b25cc5797"} Oct 30 22:30:18.566498 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.566477 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" event={"ID":"5fe7bbd0-1a6b-4524-bee5-7ee94670041e","Type":"ContainerDied","Data":"3a19e42679ccafcecb841b92d6f08a7b17071ba370c48fcbebde64d3945ee108"} Oct 30 22:30:18.566616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.566533 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx" Oct 30 22:30:18.576284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.576261 5226 scope.go:122] "RemoveContainer" containerID="3156bb9e03988616e705d9e3f79e14013de26dc8523d7a9586b80f25b6777869" Oct 30 22:30:18.773667 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.773631 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx"] Oct 30 22:30:18.779513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.779482 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29889990-mbrqx"] Oct 30 22:30:18.906658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.906584 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" path="/var/lib/kubelet/pods/57422ef2-4bae-4536-a271-52aefac80ed7/volumes" Oct 30 22:30:18.907076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.907062 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" path="/var/lib/kubelet/pods/5fe7bbd0-1a6b-4524-bee5-7ee94670041e/volumes" Oct 30 22:30:18.907513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.907500 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" path="/var/lib/kubelet/pods/ea76e61e-f47d-45b7-b679-099624d85bd1/volumes" Oct 30 22:30:18.928648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:18.928616 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb"] Oct 30 22:30:19.579951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.579910 5226 generic.go:356] "Generic (PLEG): container finished" podID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerID="add0f8a466ee4d822bb5ebdca1c8cdf9bcdfd385f96b4d6587165b0b25cc5797" exitCode=60 Oct 30 22:30:19.580344 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.580068 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" event={"ID":"9bb22783-c6ac-484c-a645-9b075a33cf60","Type":"ContainerDied","Data":"add0f8a466ee4d822bb5ebdca1c8cdf9bcdfd385f96b4d6587165b0b25cc5797"} Oct 30 22:30:19.580344 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.580171 5226 scope.go:122] "RemoveContainer" containerID="c7a29194649789404e5125b0230828b5e260994ad2071720422f857b8a45395a" Oct 30 22:30:19.580733 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.580715 5226 scope.go:122] "RemoveContainer" containerID="add0f8a466ee4d822bb5ebdca1c8cdf9bcdfd385f96b4d6587165b0b25cc5797" Oct 30 22:30:19.581112 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:19.581079 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf_odh-ai-gateway-infra(9bb22783-c6ac-484c-a645-9b075a33cf60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" Oct 30 22:30:19.760593 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.760566 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:19.866635 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.866554 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1434b45d-095f-4996-9249-b3a66a296d4d-kube-api-access-2srj5\" (UniqueName: \"kubernetes.io/projected/1434b45d-095f-4996-9249-b3a66a296d4d-kube-api-access-2srj5\") pod \"1434b45d-095f-4996-9249-b3a66a296d4d\" (UID: \"1434b45d-095f-4996-9249-b3a66a296d4d\") " Oct 30 22:30:19.866804 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.866792 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1434b45d-095f-4996-9249-b3a66a296d4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1434b45d-095f-4996-9249-b3a66a296d4d-maas-api-service-ca\") pod \"1434b45d-095f-4996-9249-b3a66a296d4d\" (UID: \"1434b45d-095f-4996-9249-b3a66a296d4d\") " Oct 30 22:30:19.867134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.867109 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1434b45d-095f-4996-9249-b3a66a296d4d-maas-api-service-ca" pod "1434b45d-095f-4996-9249-b3a66a296d4d" (UID: "1434b45d-095f-4996-9249-b3a66a296d4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.867385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.867367 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1434b45d-095f-4996-9249-b3a66a296d4d-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.868652 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.868631 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1434b45d-095f-4996-9249-b3a66a296d4d-kube-api-access-2srj5" pod "1434b45d-095f-4996-9249-b3a66a296d4d" (UID: "1434b45d-095f-4996-9249-b3a66a296d4d"). InnerVolumeSpecName "kube-api-access-2srj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.894318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.894286 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf"] Oct 30 22:30:19.968905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:19.968869 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2srj5\" (UniqueName: \"kubernetes.io/projected/1434b45d-095f-4996-9249-b3a66a296d4d-kube-api-access-2srj5\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.607437 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.607410 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" Oct 30 22:30:20.607437 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.607425 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb" event={"ID":"1434b45d-095f-4996-9249-b3a66a296d4d","Type":"ContainerDied","Data":"13bb8a15125aa6386604f565a1ec7ce46abbfe0d99bea65d247b0e88c481aa7e"} Oct 30 22:30:20.607984 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.607498 5226 scope.go:122] "RemoveContainer" containerID="56c8ff28ec4c99e1995019f26874c4e11814a6e5c26e50b883207194f94c58d0" Oct 30 22:30:20.770522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.770489 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb"] Oct 30 22:30:20.774978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.774955 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-926fc6-29889990-cnmgb"] Oct 30 22:30:20.794800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.794777 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:20.894071 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.893981 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb22783-c6ac-484c-a645-9b075a33cf60-kube-api-access-kshqf\" (UniqueName: \"kubernetes.io/projected/9bb22783-c6ac-484c-a645-9b075a33cf60-kube-api-access-kshqf\") pod \"9bb22783-c6ac-484c-a645-9b075a33cf60\" (UID: \"9bb22783-c6ac-484c-a645-9b075a33cf60\") " Oct 30 22:30:20.894270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.894149 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb22783-c6ac-484c-a645-9b075a33cf60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb22783-c6ac-484c-a645-9b075a33cf60-maas-api-service-ca\") pod \"9bb22783-c6ac-484c-a645-9b075a33cf60\" (UID: \"9bb22783-c6ac-484c-a645-9b075a33cf60\") " Oct 30 22:30:20.894795 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.894765 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb22783-c6ac-484c-a645-9b075a33cf60-maas-api-service-ca" pod "9bb22783-c6ac-484c-a645-9b075a33cf60" (UID: "9bb22783-c6ac-484c-a645-9b075a33cf60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:20.895122 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.895101 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bb22783-c6ac-484c-a645-9b075a33cf60-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:20.896360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.896334 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb22783-c6ac-484c-a645-9b075a33cf60-kube-api-access-kshqf" pod "9bb22783-c6ac-484c-a645-9b075a33cf60" (UID: "9bb22783-c6ac-484c-a645-9b075a33cf60"). InnerVolumeSpecName "kube-api-access-kshqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:20.911669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.911632 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" path="/var/lib/kubelet/pods/1434b45d-095f-4996-9249-b3a66a296d4d/volumes" Oct 30 22:30:20.997452 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:20.997419 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kshqf\" (UniqueName: \"kubernetes.io/projected/9bb22783-c6ac-484c-a645-9b075a33cf60-kube-api-access-kshqf\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:21.620520 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:21.620491 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" Oct 30 22:30:21.621035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:21.620530 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf" event={"ID":"9bb22783-c6ac-484c-a645-9b075a33cf60","Type":"ContainerDied","Data":"79e5b1af1fa9619e08bf5989c6b49f393ac3cccf6aa47787329a9eafab529748"} Oct 30 22:30:21.621035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:21.620599 5226 scope.go:122] "RemoveContainer" containerID="add0f8a466ee4d822bb5ebdca1c8cdf9bcdfd385f96b4d6587165b0b25cc5797" Oct 30 22:30:21.776477 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:21.776436 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf"] Oct 30 22:30:21.780589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:21.780559 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-29bbe735-29889990-9srvf"] Oct 30 22:30:22.906690 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:22.906642 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" path="/var/lib/kubelet/pods/9bb22783-c6ac-484c-a645-9b075a33cf60/volumes" Oct 30 22:30:24.273854 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:30:24.273818 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:30:24.421895 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:24.421858 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:30:54.412733 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:30:54.412691 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:31:07.919357 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:07.919314 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:31:08.342714 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:08.342678 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q"] Oct 30 22:31:08.446171 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:08.446138 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws"] Oct 30 22:31:13.350662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.350592 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" podUID="8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489" containerName="payload-processing" containerID="cri-o://4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9" gracePeriod=25 Oct 30 22:31:13.449419 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.449368 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" podUID="2056f2f9-5732-4c34-a715-5b7923c2cc93" containerName="payload-pre-processing" containerID="cri-o://3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354" gracePeriod=25 Oct 30 22:31:13.819396 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.819358 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:31:13.849032 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.848986 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-plugins-config-volume\") pod \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\" (UID: \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\") " Oct 30 22:31:13.849352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.849325 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-kube-api-access-bgn6x\" (UniqueName: \"kubernetes.io/projected/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-kube-api-access-bgn6x\") pod \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\" (UID: \"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489\") " Oct 30 22:31:13.849483 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.849414 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-plugins-config-volume" pod "8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489" (UID: "8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:13.850040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.850014 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:13.851507 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.851476 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-kube-api-access-bgn6x" pod "8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489" (UID: "8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489"). InnerVolumeSpecName "kube-api-access-bgn6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:13.873099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.873070 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:31:13.950796 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.950764 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2056f2f9-5732-4c34-a715-5b7923c2cc93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2056f2f9-5732-4c34-a715-5b7923c2cc93-plugins-config-volume\") pod \"2056f2f9-5732-4c34-a715-5b7923c2cc93\" (UID: \"2056f2f9-5732-4c34-a715-5b7923c2cc93\") " Oct 30 22:31:13.950987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.950817 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2056f2f9-5732-4c34-a715-5b7923c2cc93-kube-api-access-dftth\" (UniqueName: \"kubernetes.io/projected/2056f2f9-5732-4c34-a715-5b7923c2cc93-kube-api-access-dftth\") pod \"2056f2f9-5732-4c34-a715-5b7923c2cc93\" (UID: \"2056f2f9-5732-4c34-a715-5b7923c2cc93\") " Oct 30 22:31:13.951326 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.951283 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2056f2f9-5732-4c34-a715-5b7923c2cc93-plugins-config-volume" pod "2056f2f9-5732-4c34-a715-5b7923c2cc93" (UID: "2056f2f9-5732-4c34-a715-5b7923c2cc93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:13.951945 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.951918 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2056f2f9-5732-4c34-a715-5b7923c2cc93-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:13.952043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.951953 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgn6x\" (UniqueName: \"kubernetes.io/projected/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489-kube-api-access-bgn6x\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:13.953638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:13.953616 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2056f2f9-5732-4c34-a715-5b7923c2cc93-kube-api-access-dftth" pod "2056f2f9-5732-4c34-a715-5b7923c2cc93" (UID: "2056f2f9-5732-4c34-a715-5b7923c2cc93"). InnerVolumeSpecName "kube-api-access-dftth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:14.048297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.044628 5226 generic.go:356] "Generic (PLEG): container finished" podID="2056f2f9-5732-4c34-a715-5b7923c2cc93" containerID="3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354" exitCode=0 Oct 30 22:31:14.048297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.044813 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" event={"ID":"2056f2f9-5732-4c34-a715-5b7923c2cc93","Type":"ContainerDied","Data":"3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354"} Oct 30 22:31:14.048297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.044866 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" event={"ID":"2056f2f9-5732-4c34-a715-5b7923c2cc93","Type":"ContainerDied","Data":"6c3db7db42ba1bb73b01474c5f25046e324c73657aa79c18aef26ed326007e74"} Oct 30 22:31:14.048297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.044899 5226 scope.go:122] "RemoveContainer" containerID="3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354" Oct 30 22:31:14.048297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.045085 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws" Oct 30 22:31:14.065406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.065370 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dftth\" (UniqueName: \"kubernetes.io/projected/2056f2f9-5732-4c34-a715-5b7923c2cc93-kube-api-access-dftth\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:14.068869 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.068829 5226 generic.go:356] "Generic (PLEG): container finished" podID="8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489" containerID="4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9" exitCode=0 Oct 30 22:31:14.069058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.068996 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" event={"ID":"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489","Type":"ContainerDied","Data":"4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9"} Oct 30 22:31:14.069058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.069045 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" event={"ID":"8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489","Type":"ContainerDied","Data":"5d162952a75deb24f9bcdaad4310ae2de478d53f082d331ec8f43c93a60541e3"} Oct 30 22:31:14.069173 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.069120 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q" Oct 30 22:31:14.081870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.081831 5226 scope.go:122] "RemoveContainer" containerID="3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354" Oct 30 22:31:14.083158 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:31:14.083120 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354\": container with ID starting with 3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354 not found: ID does not exist" containerID="3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354" Oct 30 22:31:14.083352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.083181 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354"} err="failed to get container status \"3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354\": rpc error: code = NotFound desc = could not find container \"3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354\": container with ID starting with 3fe70087be6b892b53fdd3c77fb673346bce947a155bdecd835fdafffd362354 not found: ID does not exist" Oct 30 22:31:14.083421 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.083359 5226 scope.go:122] "RemoveContainer" containerID="4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9" Oct 30 22:31:14.102612 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.102586 5226 scope.go:122] "RemoveContainer" containerID="4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9" Oct 30 22:31:14.102952 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:31:14.102931 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9\": container with ID starting with 4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9 not found: ID does not exist" containerID="4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9" Oct 30 22:31:14.103094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.103074 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9"} err="failed to get container status \"4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9\": rpc error: code = NotFound desc = could not find container \"4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9\": container with ID starting with 4500de8cbbc7d3ccb819aeabcd47dc5795888a0b8ec9cd901fe1d97edf6bf1b9 not found: ID does not exist" Oct 30 22:31:14.240523 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.240481 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws"] Oct 30 22:31:14.253226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.253158 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61346ee8-7457f784f5-mcgws"] Oct 30 22:31:14.353473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.353436 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q"] Oct 30 22:31:14.357626 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.357598 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61346ee8-789d6d5f75-xhp6q"] Oct 30 22:31:14.905843 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.905807 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2056f2f9-5732-4c34-a715-5b7923c2cc93" path="/var/lib/kubelet/pods/2056f2f9-5732-4c34-a715-5b7923c2cc93/volumes" Oct 30 22:31:14.906405 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:14.906383 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489" path="/var/lib/kubelet/pods/8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489/volumes" Oct 30 22:31:23.702154 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:23.702117 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7"] Oct 30 22:31:23.702765 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:23.702405 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" containerID="cri-o://404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495" gracePeriod=30 Oct 30 22:31:24.421782 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:31:24.421743 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:31:28.994049 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:28.994027 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:31:29.078153 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.078099 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-envoy\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.078857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.078457 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-socket\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.078857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.078535 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-certs\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.078857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.078672 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-podinfo\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.078857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.078791 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-kube-api-access-2dhd2\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-kube-api-access-2dhd2\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.079355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.079269 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-credential-socket\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.079749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.079644 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-data\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.079907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.079864 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-token\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-token\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.080621 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.080515 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127b96df-63bc-4a6c-8bb8-f92de5603857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127b96df-63bc-4a6c-8bb8-f92de5603857-istiod-ca-cert\") pod \"127b96df-63bc-4a6c-8bb8-f92de5603857\" (UID: \"127b96df-63bc-4a6c-8bb8-f92de5603857\") " Oct 30 22:31:29.082629 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.082292 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127b96df-63bc-4a6c-8bb8-f92de5603857-istiod-ca-cert" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:29.083243 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.083064 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-data" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:29.083658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.083377 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-credential-socket" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:29.084109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.083960 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-certs" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:29.084529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.084250 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-socket" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:29.147083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.126764 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-kube-api-access-2dhd2" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "kube-api-access-2dhd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:29.147083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.138350 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-token" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:29.152581 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.152538 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-envoy" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:29.164417 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.162388 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-podinfo" pod "127b96df-63bc-4a6c-8bb8-f92de5603857" (UID: "127b96df-63bc-4a6c-8bb8-f92de5603857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:29.184132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184095 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.184132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184133 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127b96df-63bc-4a6c-8bb8-f92de5603857-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.184446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184148 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.184446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184164 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.184446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184178 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.184446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184192 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.184446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184224 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhd2\" (UniqueName: \"kubernetes.io/projected/127b96df-63bc-4a6c-8bb8-f92de5603857-kube-api-access-2dhd2\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.184446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184238 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.184446 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.184252 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127b96df-63bc-4a6c-8bb8-f92de5603857-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:29.275035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.273558 5226 generic.go:356] "Generic (PLEG): container finished" podID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerID="404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495" exitCode=0 Oct 30 22:31:29.275035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.273725 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" event={"ID":"127b96df-63bc-4a6c-8bb8-f92de5603857","Type":"ContainerDied","Data":"404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495"} Oct 30 22:31:29.275035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.273837 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" event={"ID":"127b96df-63bc-4a6c-8bb8-f92de5603857","Type":"ContainerDied","Data":"37004b194c757c47d41b3b19a27560082a71cda1c0d6cbc2fd8c9c05bcfed75a"} Oct 30 22:31:29.275035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.273882 5226 scope.go:122] "RemoveContainer" containerID="404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495" Oct 30 22:31:29.275035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.273935 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7" Oct 30 22:31:29.294917 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.294890 5226 scope.go:122] "RemoveContainer" containerID="404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495" Oct 30 22:31:29.295379 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:31:29.295354 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495\": container with ID starting with 404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495 not found: ID does not exist" containerID="404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495" Oct 30 22:31:29.295476 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.295392 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495"} err="failed to get container status \"404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495\": rpc error: code = NotFound desc = could not find container \"404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495\": container with ID starting with 404f4c52ff31fdce1cb5aa5c5095d51fc8e3046ccaab25e3f188c17e42b4a495 not found: ID does not exist" Oct 30 22:31:29.442224 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.442177 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7"] Oct 30 22:31:29.446649 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:29.446587 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-4kps7"] Oct 30 22:31:30.905636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:30.905605 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" path="/var/lib/kubelet/pods/127b96df-63bc-4a6c-8bb8-f92de5603857/volumes" Oct 30 22:31:38.613267 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:38.613231 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw"] Oct 30 22:31:38.613798 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:38.613474 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" containerID="cri-o://67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39" gracePeriod=30 Oct 30 22:31:39.249541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:39.249506 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9"] Oct 30 22:31:39.249849 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:39.249753 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" containerID="cri-o://617c696bb9618977a9d588705f825780749a77f3c67569d2af91dd6d8aa75073" gracePeriod=30 Oct 30 22:31:39.413214 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:39.413173 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8"] Oct 30 22:31:39.413468 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:39.413408 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" containerID="cri-o://ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808" gracePeriod=30 Oct 30 22:31:43.887263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:43.887180 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:31:44.044287 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044252 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-token\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-token\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.044447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044345 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-socket\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.044447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044373 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-podinfo\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.044447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044398 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-envoy\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.044447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044421 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-data\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.044619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044521 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-certs\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.044677 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044657 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-kube-api-access-w7ng2\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-kube-api-access-w7ng2\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.044738 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044713 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-socket" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.044797 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044772 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-data" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.044976 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044798 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25edfa7-7bde-4b24-8ff8-306783a3f158-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25edfa7-7bde-4b24-8ff8-306783a3f158-istiod-ca-cert\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.044976 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044816 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-certs" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.044976 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.044844 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-credential-socket\") pod \"c25edfa7-7bde-4b24-8ff8-306783a3f158\" (UID: \"c25edfa7-7bde-4b24-8ff8-306783a3f158\") " Oct 30 22:31:44.045163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.045053 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25edfa7-7bde-4b24-8ff8-306783a3f158-istiod-ca-cert" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:44.045163 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.045083 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-credential-socket" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.045860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.045836 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.045967 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.045863 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.045967 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.045878 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.045967 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.045892 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c25edfa7-7bde-4b24-8ff8-306783a3f158-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.045967 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.045906 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.047306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.047266 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-envoy" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.047306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.047271 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-token" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.047450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.047331 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-kube-api-access-w7ng2" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "kube-api-access-w7ng2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.047450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.047347 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-podinfo" pod "c25edfa7-7bde-4b24-8ff8-306783a3f158" (UID: "c25edfa7-7bde-4b24-8ff8-306783a3f158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:44.145889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.145786 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:31:44.146067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.145882 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:31:44.147327 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.147298 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7ng2\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-kube-api-access-w7ng2\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.147445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.147331 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.147445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.147347 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.147445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.147363 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c25edfa7-7bde-4b24-8ff8-306783a3f158-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.494156 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.494118 5226 generic.go:356] "Generic (PLEG): container finished" podID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerID="617c696bb9618977a9d588705f825780749a77f3c67569d2af91dd6d8aa75073" exitCode=0 Oct 30 22:31:44.494352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.494269 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" event={"ID":"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2","Type":"ContainerDied","Data":"617c696bb9618977a9d588705f825780749a77f3c67569d2af91dd6d8aa75073"} Oct 30 22:31:44.496382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.496265 5226 generic.go:356] "Generic (PLEG): container finished" podID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerID="67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39" exitCode=0 Oct 30 22:31:44.496531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.496399 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" event={"ID":"c25edfa7-7bde-4b24-8ff8-306783a3f158","Type":"ContainerDied","Data":"67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39"} Oct 30 22:31:44.496531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.496436 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" event={"ID":"c25edfa7-7bde-4b24-8ff8-306783a3f158","Type":"ContainerDied","Data":"1267f02d0028d824f00c3f67928933e0d7734da6613742001c55ed7906043807"} Oct 30 22:31:44.496531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.496460 5226 scope.go:122] "RemoveContainer" containerID="67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39" Oct 30 22:31:44.496711 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.496601 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw" Oct 30 22:31:44.511482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.511464 5226 scope.go:122] "RemoveContainer" containerID="67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39" Oct 30 22:31:44.511875 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:31:44.511832 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39\": container with ID starting with 67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39 not found: ID does not exist" containerID="67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39" Oct 30 22:31:44.512018 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.511877 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39"} err="failed to get container status \"67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39\": rpc error: code = NotFound desc = could not find container \"67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39\": container with ID starting with 67194d90f9e5989fac2fd45eef74e195b2c4fb42e40246065c47e0530cb55e39 not found: ID does not exist" Oct 30 22:31:44.682242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.674613 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw"] Oct 30 22:31:44.693498 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.693449 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-79vbw"] Oct 30 22:31:44.801598 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.801562 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:31:44.860425 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.860398 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:31:44.879335 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879302 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-certs\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.879525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879374 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-data\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.879525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879399 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-envoy\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.879525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879454 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-credential-socket\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.879525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879485 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-token\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-token\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.879746 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879567 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-envoy\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.879746 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879628 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-token\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-token\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.879746 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879639 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-certs" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.879746 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879697 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-credential-socket\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.879959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879747 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-podinfo\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.879959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879697 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-data" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.879959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879789 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istiod-ca-cert\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.879959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879852 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-socket\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.879959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879785 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-credential-socket" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.879959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879886 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-certs\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.879959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879921 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-credential-socket" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.879959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879930 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-podinfo\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.880316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.879969 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-socket\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.880316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.880000 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-kube-api-access-bsp6b\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-kube-api-access-bsp6b\") pod \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\" (UID: \"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2\") " Oct 30 22:31:44.880316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.880084 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-data\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.880316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.880111 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-socket" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.880316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.880198 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f6c12a-6677-4996-a7b9-fa3c581e7873-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f6c12a-6677-4996-a7b9-fa3c581e7873-istiod-ca-cert\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.880316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.880243 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-kube-api-access-xp57g\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-kube-api-access-xp57g\") pod \"42f6c12a-6677-4996-a7b9-fa3c581e7873\" (UID: \"42f6c12a-6677-4996-a7b9-fa3c581e7873\") " Oct 30 22:31:44.880653 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.880536 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istiod-ca-cert" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:44.880818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.880794 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-data" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.881135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881113 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.881135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881123 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f6c12a-6677-4996-a7b9-fa3c581e7873-istiod-ca-cert" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:44.881328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881140 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.881328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881155 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.881328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881169 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.881328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881183 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.881328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881196 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.881328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881235 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.881689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.881418 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-socket" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.882584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.882556 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-envoy" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.882817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.882796 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-certs" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.882927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.882893 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-envoy" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:44.883179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.882948 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-token" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.883493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.883473 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-kube-api-access-xp57g" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "kube-api-access-xp57g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.883681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.883658 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-podinfo" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:44.883961 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.883816 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-kube-api-access-bsp6b" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "kube-api-access-bsp6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.883961 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.883929 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-token" pod "42f6c12a-6677-4996-a7b9-fa3c581e7873" (UID: "42f6c12a-6677-4996-a7b9-fa3c581e7873"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:44.884991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.884935 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-podinfo" pod "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" (UID: "c5a279b6-59b5-4ccc-8f77-8589e7eac9f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:44.932303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.914177 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" path="/var/lib/kubelet/pods/c25edfa7-7bde-4b24-8ff8-306783a3f158/volumes" Oct 30 22:31:44.984794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.984757 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42f6c12a-6677-4996-a7b9-fa3c581e7873-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.984992 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp57g\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-kube-api-access-xp57g\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985128 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985250 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985389 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985376 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985484 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985588 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985677 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42f6c12a-6677-4996-a7b9-fa3c581e7873-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985777 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42f6c12a-6677-4996-a7b9-fa3c581e7873-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985863 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:44.985958 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:44.985947 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsp6b\" (UniqueName: \"kubernetes.io/projected/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2-kube-api-access-bsp6b\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:45.537196 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.537149 5226 generic.go:356] "Generic (PLEG): container finished" podID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerID="ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808" exitCode=0 Oct 30 22:31:45.537553 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.537295 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" Oct 30 22:31:45.538373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.537312 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" event={"ID":"42f6c12a-6677-4996-a7b9-fa3c581e7873","Type":"ContainerDied","Data":"ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808"} Oct 30 22:31:45.538373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.538227 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8" event={"ID":"42f6c12a-6677-4996-a7b9-fa3c581e7873","Type":"ContainerDied","Data":"a56448d51b0e8ea6439aaf28b91a0873bb498eb663b9d7e799c7674e6add3cf2"} Oct 30 22:31:45.538373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.538271 5226 scope.go:122] "RemoveContainer" containerID="ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808" Oct 30 22:31:45.539956 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.539931 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" event={"ID":"c5a279b6-59b5-4ccc-8f77-8589e7eac9f2","Type":"ContainerDied","Data":"c7938ed6347a6fc98487f8ace0cd9d9c60def24fbc4d905827676306ba12975b"} Oct 30 22:31:45.540079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.539941 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9" Oct 30 22:31:45.549681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.549662 5226 scope.go:122] "RemoveContainer" containerID="ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808" Oct 30 22:31:45.549948 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:31:45.549929 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808\": container with ID starting with ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808 not found: ID does not exist" containerID="ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808" Oct 30 22:31:45.550003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.549956 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808"} err="failed to get container status \"ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808\": rpc error: code = NotFound desc = could not find container \"ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808\": container with ID starting with ff0111b9846bf84e07dfe6c3bc05233d188edc8194f678e03812c79dadd68808 not found: ID does not exist" Oct 30 22:31:45.550003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.549973 5226 scope.go:122] "RemoveContainer" containerID="617c696bb9618977a9d588705f825780749a77f3c67569d2af91dd6d8aa75073" Oct 30 22:31:45.695366 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.695324 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8"] Oct 30 22:31:45.746309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.746258 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-ljbn8"] Oct 30 22:31:45.808821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.808695 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9"] Oct 30 22:31:45.813517 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:45.813477 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-49zd9"] Oct 30 22:31:46.906009 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:46.905977 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" path="/var/lib/kubelet/pods/42f6c12a-6677-4996-a7b9-fa3c581e7873/volumes" Oct 30 22:31:46.906588 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:46.906571 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" path="/var/lib/kubelet/pods/c5a279b6-59b5-4ccc-8f77-8589e7eac9f2/volumes" Oct 30 22:31:49.278274 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:49.278237 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:31:49.291240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:49.291196 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:31:54.414588 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:31:54.414545 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:31:58.574787 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:58.574750 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc"] Oct 30 22:31:58.575191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:58.574981 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" podUID="f7f12682-fb05-4d24-867f-1866d54d928a" containerName="maas-api" containerID="cri-o://286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6" gracePeriod=30 Oct 30 22:31:58.972497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:58.972455 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:31:59.124279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.122826 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls\") pod \"f7f12682-fb05-4d24-867f-1866d54d928a\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " Oct 30 22:31:59.124279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.122914 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f12682-fb05-4d24-867f-1866d54d928a-kube-api-access-rrwsl\" (UniqueName: \"kubernetes.io/projected/f7f12682-fb05-4d24-867f-1866d54d928a-kube-api-access-rrwsl\") pod \"f7f12682-fb05-4d24-867f-1866d54d928a\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " Oct 30 22:31:59.124279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.123109 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-metrics-certs\") pod \"f7f12682-fb05-4d24-867f-1866d54d928a\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " Oct 30 22:31:59.124279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.123314 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f12682-fb05-4d24-867f-1866d54d928a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f12682-fb05-4d24-867f-1866d54d928a-openshift-service-ca\") pod \"f7f12682-fb05-4d24-867f-1866d54d928a\" (UID: \"f7f12682-fb05-4d24-867f-1866d54d928a\") " Oct 30 22:31:59.124622 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.124456 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f12682-fb05-4d24-867f-1866d54d928a-openshift-service-ca" pod "f7f12682-fb05-4d24-867f-1866d54d928a" (UID: "f7f12682-fb05-4d24-867f-1866d54d928a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:59.133367 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.133326 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-metrics-certs" pod "f7f12682-fb05-4d24-867f-1866d54d928a" (UID: "f7f12682-fb05-4d24-867f-1866d54d928a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:59.134519 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.134489 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f12682-fb05-4d24-867f-1866d54d928a-kube-api-access-rrwsl" pod "f7f12682-fb05-4d24-867f-1866d54d928a" (UID: "f7f12682-fb05-4d24-867f-1866d54d928a"). InnerVolumeSpecName "kube-api-access-rrwsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:59.134948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.134924 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls" pod "f7f12682-fb05-4d24-867f-1866d54d928a" (UID: "f7f12682-fb05-4d24-867f-1866d54d928a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:59.225926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.225842 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f12682-fb05-4d24-867f-1866d54d928a-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.225926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.225885 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.225926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.225899 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwsl\" (UniqueName: \"kubernetes.io/projected/f7f12682-fb05-4d24-867f-1866d54d928a-kube-api-access-rrwsl\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.225926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.225911 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f12682-fb05-4d24-867f-1866d54d928a-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:59.658433 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.658347 5226 generic.go:356] "Generic (PLEG): container finished" podID="f7f12682-fb05-4d24-867f-1866d54d928a" containerID="286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6" exitCode=0 Oct 30 22:31:59.658433 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.658396 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" Oct 30 22:31:59.658936 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.658427 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" event={"ID":"f7f12682-fb05-4d24-867f-1866d54d928a","Type":"ContainerDied","Data":"286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6"} Oct 30 22:31:59.658936 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.658480 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc" event={"ID":"f7f12682-fb05-4d24-867f-1866d54d928a","Type":"ContainerDied","Data":"69b45d7ca4c4c8fca62bcd54d26c4d4309c2b607552909190db9f20a741f5ede"} Oct 30 22:31:59.658936 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.658501 5226 scope.go:122] "RemoveContainer" containerID="286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6" Oct 30 22:31:59.670251 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.670227 5226 scope.go:122] "RemoveContainer" containerID="286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6" Oct 30 22:31:59.670550 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:31:59.670526 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6\": container with ID starting with 286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6 not found: ID does not exist" containerID="286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6" Oct 30 22:31:59.670599 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.670559 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6"} err="failed to get container status \"286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6\": rpc error: code = NotFound desc = could not find container \"286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6\": container with ID starting with 286d21079ea742321c04128783461750430d3388eba4bed50c4e8a5d788e0bd6 not found: ID does not exist" Oct 30 22:31:59.816262 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.816228 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc"] Oct 30 22:31:59.853821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:31:59.853781 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61346ee8-76d898f7b8-vg7pc"] Oct 30 22:32:00.910460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:00.910423 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f12682-fb05-4d24-867f-1866d54d928a" path="/var/lib/kubelet/pods/f7f12682-fb05-4d24-867f-1866d54d928a/volumes" Oct 30 22:32:24.412503 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:32:24.412465 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:32:28.996109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:28.996073 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:32:51.547294 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.547252 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf"] Oct 30 22:32:51.548091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548068 5226 cpu_manager.go:414] "containerMap: removing container" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548092 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548106 5226 cpu_manager.go:414] "containerMap: removing container" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548113 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548129 5226 cpu_manager.go:414] "containerMap: removing container" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548136 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548150 5226 cpu_manager.go:414] "containerMap: removing container" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548158 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548175 5226 cpu_manager.go:414] "containerMap: removing container" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548181 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548192 5226 cpu_manager.go:414] "containerMap: removing container" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548199 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548229 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.548232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548236 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548247 5226 cpu_manager.go:414] "containerMap: removing container" podUID="f7f12682-fb05-4d24-867f-1866d54d928a" containerName="maas-api" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548254 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f12682-fb05-4d24-867f-1866d54d928a" containerName="maas-api" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548265 5226 cpu_manager.go:414] "containerMap: removing container" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548271 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548284 5226 cpu_manager.go:414] "containerMap: removing container" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548291 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548303 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548312 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548330 5226 cpu_manager.go:414] "containerMap: removing container" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548337 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548349 5226 cpu_manager.go:414] "containerMap: removing container" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548356 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548374 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548382 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548394 5226 cpu_manager.go:414] "containerMap: removing container" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548402 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548416 5226 cpu_manager.go:414] "containerMap: removing container" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548423 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548449 5226 cpu_manager.go:414] "containerMap: removing container" podUID="2056f2f9-5732-4c34-a715-5b7923c2cc93" containerName="payload-pre-processing" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548456 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2056f2f9-5732-4c34-a715-5b7923c2cc93" containerName="payload-pre-processing" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548471 5226 cpu_manager.go:414] "containerMap: removing container" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548479 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548500 5226 cpu_manager.go:414] "containerMap: removing container" podUID="8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489" containerName="payload-processing" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548508 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489" containerName="payload-processing" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548517 5226 cpu_manager.go:414] "containerMap: removing container" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548524 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548687 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548700 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="127b96df-63bc-4a6c-8bb8-f92de5603857" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548712 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a279b6-59b5-4ccc-8f77-8589e7eac9f2" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548722 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548732 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548742 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="2056f2f9-5732-4c34-a715-5b7923c2cc93" containerName="payload-pre-processing" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548753 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548765 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee4c7ba-6ed4-475f-9e44-6fdcea8a7489" containerName="payload-processing" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548775 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548785 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548793 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548803 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548812 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548819 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25edfa7-7bde-4b24-8ff8-306783a3f158" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548832 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548840 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f12682-fb05-4d24-867f-1866d54d928a" containerName="maas-api" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548847 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f6c12a-6677-4996-a7b9-fa3c581e7873" containerName="istio-proxy" Oct 30 22:32:51.548883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.548856 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea76e61e-f47d-45b7-b679-099624d85bd1" containerName="cleanup" Oct 30 22:32:51.554544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.554523 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.604902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.604871 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c3c2cf-serving-cert\"" Oct 30 22:32:51.617784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.617756 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.617945 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.617878 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-metrics-certs\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.617994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.617943 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.618039 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.618004 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f29lk\" (UniqueName: \"kubernetes.io/projected/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-kube-api-access-f29lk\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.654656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.654620 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf"] Oct 30 22:32:51.719717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.719681 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.719905 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:32:51.719832 5226 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-serving-cert: secret "maas-api-e2e-inf-a-c3c2cf-serving-cert" not found Oct 30 22:32:51.719905 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:32:51.719899 5226 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls podName:5c6e8a7f-7402-45bf-bfd4-0db00d1c1270 nodeName:}" failed. No retries permitted until 2026-10-30 22:32:52.219880327 +0000 UTC m=+1863.477222015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls") pod "maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" (UID: "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270") : secret "maas-api-e2e-inf-a-c3c2cf-serving-cert" not found Oct 30 22:32:51.720040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.719830 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-metrics-certs\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.720040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.719969 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.720040 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.720022 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f29lk\" (UniqueName: \"kubernetes.io/projected/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-kube-api-access-f29lk\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.720621 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.720602 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.722257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.722227 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-metrics-certs\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:51.777890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:51.777856 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f29lk\" (UniqueName: \"kubernetes.io/projected/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-kube-api-access-f29lk\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:52.227343 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:52.227309 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:52.229814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:52.229789 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls\") pod \"maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:52.469120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:52.469083 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:52.845606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:52.845457 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf"] Oct 30 22:32:52.850912 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:32:52.850883 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6e8a7f_7402_45bf_bfd4_0db00d1c1270.slice/crio-62c7f595a901ab8d65813cd2126e1fa6a72a5694d7dc6da0d074930e31479282 WatchSource:0}: Error finding container 62c7f595a901ab8d65813cd2126e1fa6a72a5694d7dc6da0d074930e31479282: Status 404 returned error can't find the container with id 62c7f595a901ab8d65813cd2126e1fa6a72a5694d7dc6da0d074930e31479282 Oct 30 22:32:52.852452 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:52.852427 5226 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:32:53.044490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:53.044459 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" event={"ID":"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270","Type":"ContainerStarted","Data":"62c7f595a901ab8d65813cd2126e1fa6a72a5694d7dc6da0d074930e31479282"} Oct 30 22:32:54.053616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:54.053524 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" event={"ID":"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270","Type":"ContainerStarted","Data":"f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c"} Oct 30 22:32:54.054045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:54.053645 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:32:54.163707 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:32:54.163646 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" podStartSLOduration=2.232930216 podStartE2EDuration="3.163629355s" totalImagesPullingTime="930.699139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:54.163038936 +0000 UTC m=+1865.420380645" watchObservedRunningTime="2026-10-30 22:32:54.163629355 +0000 UTC m=+1865.420971067" Oct 30 22:32:54.413169 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:32:54.413073 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:33:00.066960 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:00.066923 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:33:14.832366 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.832324 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9"] Oct 30 22:33:14.833186 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.833160 5226 cpu_manager.go:414] "containerMap: removing container" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:33:14.833346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.833188 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:33:14.833346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.833230 5226 cpu_manager.go:414] "containerMap: removing container" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:33:14.833346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.833239 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:33:14.833506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.833449 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="1434b45d-095f-4996-9249-b3a66a296d4d" containerName="cleanup" Oct 30 22:33:14.833506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.833471 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="57422ef2-4bae-4536-a271-52aefac80ed7" containerName="cleanup" Oct 30 22:33:14.833506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.833491 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb22783-c6ac-484c-a645-9b075a33cf60" containerName="cleanup" Oct 30 22:33:14.833506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.833502 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe7bbd0-1a6b-4524-bee5-7ee94670041e" containerName="cleanup" Oct 30 22:33:14.836905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.836882 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:14.885617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.885581 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d57980-serving-cert\"" Oct 30 22:33:14.937369 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.937329 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9"] Oct 30 22:33:14.976530 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.976474 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:14.976788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.976752 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:14.976892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.976809 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wg6\" (UniqueName: \"kubernetes.io/projected/45dcc87b-99f4-4f75-851c-2bd67975600d-kube-api-access-x5wg6\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:14.976991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:14.976967 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dcc87b-99f4-4f75-851c-2bd67975600d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.087393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.087301 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.087393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.087350 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wg6\" (UniqueName: \"kubernetes.io/projected/45dcc87b-99f4-4f75-851c-2bd67975600d-kube-api-access-x5wg6\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.087637 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.087475 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dcc87b-99f4-4f75-851c-2bd67975600d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.087739 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.087718 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.087950 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:33:15.087893 5226 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-serving-cert: secret "maas-api-e2e-shared-b-w3-d57980-serving-cert" not found Oct 30 22:33:15.088030 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:33:15.087957 5226 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls podName:45dcc87b-99f4-4f75-851c-2bd67975600d nodeName:}" failed. No retries permitted until 2026-10-30 22:33:15.587935954 +0000 UTC m=+1886.845277641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" (UID: "45dcc87b-99f4-4f75-851c-2bd67975600d") : secret "maas-api-e2e-shared-b-w3-d57980-serving-cert" not found Oct 30 22:33:15.088944 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.088918 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dcc87b-99f4-4f75-851c-2bd67975600d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.092132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.092100 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.147573 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.147535 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wg6\" (UniqueName: \"kubernetes.io/projected/45dcc87b-99f4-4f75-851c-2bd67975600d-kube-api-access-x5wg6\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.599130 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.599093 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.601590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.601564 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:15.751229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:15.751106 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:16.186757 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:16.186702 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9"] Oct 30 22:33:16.191521 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:33:16.191486 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45dcc87b_99f4_4f75_851c_2bd67975600d.slice/crio-f311e168ca2089ddd5f9e742773cf5d200ec037c3883ffd1c7884ad5291d3c7d WatchSource:0}: Error finding container f311e168ca2089ddd5f9e742773cf5d200ec037c3883ffd1c7884ad5291d3c7d: Status 404 returned error can't find the container with id f311e168ca2089ddd5f9e742773cf5d200ec037c3883ffd1c7884ad5291d3c7d Oct 30 22:33:16.274480 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:16.274438 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" event={"ID":"45dcc87b-99f4-4f75-851c-2bd67975600d","Type":"ContainerStarted","Data":"f311e168ca2089ddd5f9e742773cf5d200ec037c3883ffd1c7884ad5291d3c7d"} Oct 30 22:33:18.411227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:18.410259 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" event={"ID":"45dcc87b-99f4-4f75-851c-2bd67975600d","Type":"ContainerStarted","Data":"0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74"} Oct 30 22:33:18.411716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:18.411333 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:18.529228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:18.525471 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" podStartSLOduration=3.5546445909999997 podStartE2EDuration="4.525452014s" totalImagesPullingTime="970.807423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:18.523613646 +0000 UTC m=+1889.780955381" watchObservedRunningTime="2026-10-30 22:33:18.525452014 +0000 UTC m=+1889.782793725" Oct 30 22:33:24.411846 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:33:24.411804 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:33:25.435772 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:33:25.435742 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:33:54.412481 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:33:54.412442 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:34:02.511239 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:02.511175 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:34:11.284895 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:11.284851 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6"] Oct 30 22:34:11.285553 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:11.285171 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" containerID="cri-o://b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e" gracePeriod=30 Oct 30 22:34:13.290166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:13.290118 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:34:13.290564 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:13.290178 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:34:16.565559 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.565500 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:34:16.646757 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.646731 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-data\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.646908 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.646838 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-certs\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.646981 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.646936 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istiod-ca-cert\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.646981 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.646956 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-credential-socket\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.647095 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.646983 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-podinfo\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.647095 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.646985 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-data" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.647095 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647011 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-token\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-token\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.647095 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647087 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-envoy\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.647318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647132 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-certs" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.647318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647174 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-kube-api-access-99tq4\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-kube-api-access-99tq4\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.647318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647245 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-socket\") pod \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\" (UID: \"0cd77cda-1cff-4e39-8f54-93097fa58c1c\") " Oct 30 22:34:16.647318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647293 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-credential-socket" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.647487 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647387 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istiod-ca-cert" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:16.647635 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647615 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-socket" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.647858 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647836 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.647911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647861 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.647911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647876 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.647911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647890 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.647911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.647902 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.649728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.649699 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-podinfo" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:16.649841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.649759 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-envoy" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.649841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.649779 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-kube-api-access-99tq4" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "kube-api-access-99tq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:16.649841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.649799 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-token" pod "0cd77cda-1cff-4e39-8f54-93097fa58c1c" (UID: "0cd77cda-1cff-4e39-8f54-93097fa58c1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:16.749043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.749005 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.749043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.749037 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tq4\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-kube-api-access-99tq4\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.749043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.749049 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.749465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.749058 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd77cda-1cff-4e39-8f54-93097fa58c1c-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.848298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.842805 5226 generic.go:356] "Generic (PLEG): container finished" podID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerID="b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e" exitCode=0 Oct 30 22:34:16.848298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.843113 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" event={"ID":"0cd77cda-1cff-4e39-8f54-93097fa58c1c","Type":"ContainerDied","Data":"b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e"} Oct 30 22:34:16.848298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.843231 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" event={"ID":"0cd77cda-1cff-4e39-8f54-93097fa58c1c","Type":"ContainerDied","Data":"9075396990de1f5bcc1ea9ea67ebb4d9ebfd2446fc78fca0f6f374d8b3b348f3"} Oct 30 22:34:16.848298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.843273 5226 scope.go:122] "RemoveContainer" containerID="b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e" Oct 30 22:34:16.848298 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.843526 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6" Oct 30 22:34:16.858213 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.858180 5226 scope.go:122] "RemoveContainer" containerID="b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e" Oct 30 22:34:16.858520 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:34:16.858497 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e\": container with ID starting with b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e not found: ID does not exist" containerID="b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e" Oct 30 22:34:16.858601 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:16.858528 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e"} err="failed to get container status \"b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e\": rpc error: code = NotFound desc = could not find container \"b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e\": container with ID starting with b467a586321f552ed45775695b589616df63d07d72a76edaec45f65cc6e0b84e not found: ID does not exist" Oct 30 22:34:17.011463 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:17.011431 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6"] Oct 30 22:34:17.015215 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:17.015169 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-88jz6"] Oct 30 22:34:18.906438 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:18.906409 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" path="/var/lib/kubelet/pods/0cd77cda-1cff-4e39-8f54-93097fa58c1c/volumes" Oct 30 22:34:24.411064 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:34:24.411015 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:34:24.688871 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:24.688838 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x"] Oct 30 22:34:24.689114 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:24.689092 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" containerID="cri-o://c30ab5fcef77f84a1bf83e8b63aef01cfad23b0cda7518c8135490115c768d45" gracePeriod=30 Oct 30 22:34:24.735157 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:24.735123 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp"] Oct 30 22:34:24.735401 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:24.735364 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" containerID="cri-o://4448910437af5e2b64a165cce513a1145a797506213b043bd7700ff2a1653642" gracePeriod=30 Oct 30 22:34:25.509189 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:25.509150 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:34:25.509651 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:25.509243 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:34:25.528715 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:25.528683 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl"] Oct 30 22:34:25.528907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:25.528883 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" containerID="cri-o://cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75" gracePeriod=30 Oct 30 22:34:26.366732 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:26.366698 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:34:26.366906 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:26.366749 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:34:29.382174 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:29.382135 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:34:29.382560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:29.382218 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:34:29.939297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:29.939266 5226 generic.go:356] "Generic (PLEG): container finished" podID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerID="c30ab5fcef77f84a1bf83e8b63aef01cfad23b0cda7518c8135490115c768d45" exitCode=0 Oct 30 22:34:29.939459 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:29.939322 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" event={"ID":"f8c2a7ce-218e-4096-a541-f016e8f53f00","Type":"ContainerDied","Data":"c30ab5fcef77f84a1bf83e8b63aef01cfad23b0cda7518c8135490115c768d45"} Oct 30 22:34:29.941061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:29.941035 5226 generic.go:356] "Generic (PLEG): container finished" podID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerID="4448910437af5e2b64a165cce513a1145a797506213b043bd7700ff2a1653642" exitCode=0 Oct 30 22:34:29.941195 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:29.941105 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" event={"ID":"3eb179ec-ed21-46d3-88e4-45185d9e22d7","Type":"ContainerDied","Data":"4448910437af5e2b64a165cce513a1145a797506213b043bd7700ff2a1653642"} Oct 30 22:34:30.022683 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.022660 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:34:30.047431 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047394 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-podinfo\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.047575 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047450 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-envoy\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.047575 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047534 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-kube-api-access-qtqxl\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-kube-api-access-qtqxl\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.047687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047609 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-credential-socket\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.047741 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047703 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-data\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.047834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047820 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-socket\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.047896 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047879 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c2a7ce-218e-4096-a541-f016e8f53f00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2a7ce-218e-4096-a541-f016e8f53f00-istiod-ca-cert\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.047950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047881 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-credential-socket" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.047950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047923 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-token\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.048066 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047959 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-data" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.048066 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.047988 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-certs\") pod \"f8c2a7ce-218e-4096-a541-f016e8f53f00\" (UID: \"f8c2a7ce-218e-4096-a541-f016e8f53f00\") " Oct 30 22:34:30.048173 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.048086 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-socket" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.048443 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.048417 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c2a7ce-218e-4096-a541-f016e8f53f00-istiod-ca-cert" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:30.048598 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.048567 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-certs" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.048664 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.048598 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.048664 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.048622 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2a7ce-218e-4096-a541-f016e8f53f00-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.048664 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.048637 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.048664 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.048651 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.050085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.050062 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-envoy" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.050167 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.050102 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-token" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:30.050167 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.050130 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-kube-api-access-qtqxl" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "kube-api-access-qtqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:30.050269 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.050163 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-podinfo" pod "f8c2a7ce-218e-4096-a541-f016e8f53f00" (UID: "f8c2a7ce-218e-4096-a541-f016e8f53f00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:30.068092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.068071 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:34:30.149195 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149162 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-kube-api-access-mnxml\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-kube-api-access-mnxml\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149248 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-credential-socket\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149350 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-envoy\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149383 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-token\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-token\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149410 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istiod-ca-cert\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149473 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-data\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149501 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-certs\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149498 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-credential-socket" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.149694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149542 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-socket\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149637 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-podinfo\") pod \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\" (UID: \"3eb179ec-ed21-46d3-88e4-45185d9e22d7\") " Oct 30 22:34:30.149884 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149800 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-data" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.149884 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.149829 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istiod-ca-cert" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:30.150091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150051 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-certs" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.150295 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150273 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-socket" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.150502 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150476 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150561 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150511 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150561 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150528 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtqxl\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-kube-api-access-qtqxl\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150561 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150545 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150559 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150576 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150592 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2a7ce-218e-4096-a541-f016e8f53f00-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150605 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2a7ce-218e-4096-a541-f016e8f53f00-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150619 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.150709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.150633 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.151504 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.151477 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-kube-api-access-mnxml" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "kube-api-access-mnxml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:30.151644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.151626 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-envoy" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.151897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.151883 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-token" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:30.151962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.151949 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-podinfo" pod "3eb179ec-ed21-46d3-88e4-45185d9e22d7" (UID: "3eb179ec-ed21-46d3-88e4-45185d9e22d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:30.252769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.252679 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.252769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.252715 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.252769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.252729 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb179ec-ed21-46d3-88e4-45185d9e22d7-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.252769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.252744 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnxml\" (UniqueName: \"kubernetes.io/projected/3eb179ec-ed21-46d3-88e4-45185d9e22d7-kube-api-access-mnxml\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.806313 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.806251 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:34:30.860951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.860921 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istiod-ca-cert\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.860964 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-data\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861026 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-socket\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861058 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-kube-api-access-zfxbd\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-kube-api-access-zfxbd\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861087 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-envoy\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861163 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-certs\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861251 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-credential-socket\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861286 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-token\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-token\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861322 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-podinfo\") pod \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\" (UID: \"5678dc0a-2ca6-41ce-852f-064f3566ce4b\") " Oct 30 22:34:30.861566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861355 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-socket" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.861566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861383 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-data" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.861566 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861408 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istiod-ca-cert" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:30.861737 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861572 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-credential-socket" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.861737 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.861675 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-certs" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.862174 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.862152 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.862302 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.862180 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.862302 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.862197 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.862302 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.862227 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.862302 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.862244 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.863263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.863239 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-envoy" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:30.863441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.863426 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-token" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:30.863487 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.863453 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-podinfo" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:30.863540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.863526 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-kube-api-access-zfxbd" pod "5678dc0a-2ca6-41ce-852f-064f3566ce4b" (UID: "5678dc0a-2ca6-41ce-852f-064f3566ce4b"). InnerVolumeSpecName "kube-api-access-zfxbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:30.949973 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.949936 5226 generic.go:356] "Generic (PLEG): container finished" podID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerID="cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75" exitCode=0 Oct 30 22:34:30.950164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.949988 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" Oct 30 22:34:30.950164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.950029 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" event={"ID":"5678dc0a-2ca6-41ce-852f-064f3566ce4b","Type":"ContainerDied","Data":"cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75"} Oct 30 22:34:30.950164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.950096 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl" event={"ID":"5678dc0a-2ca6-41ce-852f-064f3566ce4b","Type":"ContainerDied","Data":"d893826905a8d4137beb83c7a6c4bbc8cbf96a98716ee136596cb2a44f3ab1a5"} Oct 30 22:34:30.950164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.950130 5226 scope.go:122] "RemoveContainer" containerID="cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75" Oct 30 22:34:30.952794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.952774 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" Oct 30 22:34:30.953447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.952810 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x" event={"ID":"f8c2a7ce-218e-4096-a541-f016e8f53f00","Type":"ContainerDied","Data":"87a30642d7d5023f209f231eb0301c386c21a1447c24dcb95935bc820aeb76e3"} Oct 30 22:34:30.955160 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.954798 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" Oct 30 22:34:30.955160 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.954829 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp" event={"ID":"3eb179ec-ed21-46d3-88e4-45185d9e22d7","Type":"ContainerDied","Data":"12571f37df2725da6c42a21500ba98fe31f4e4655fd54ac04d48e86dc95e4208"} Oct 30 22:34:30.963092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.962984 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfxbd\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-kube-api-access-zfxbd\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.963092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.963011 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.963092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.963025 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.963092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.963034 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5678dc0a-2ca6-41ce-852f-064f3566ce4b-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:30.963470 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.963154 5226 scope.go:122] "RemoveContainer" containerID="cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75" Oct 30 22:34:30.963520 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:34:30.963484 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75\": container with ID starting with cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75 not found: ID does not exist" containerID="cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75" Oct 30 22:34:30.963520 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.963510 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75"} err="failed to get container status \"cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75\": rpc error: code = NotFound desc = could not find container \"cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75\": container with ID starting with cb4d3d01e175a43498d1dfe80f7489a151a38275e8fa512bb3c1c62010b9db75 not found: ID does not exist" Oct 30 22:34:30.963622 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.963527 5226 scope.go:122] "RemoveContainer" containerID="c30ab5fcef77f84a1bf83e8b63aef01cfad23b0cda7518c8135490115c768d45" Oct 30 22:34:30.975544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:30.975524 5226 scope.go:122] "RemoveContainer" containerID="4448910437af5e2b64a165cce513a1145a797506213b043bd7700ff2a1653642" Oct 30 22:34:31.129134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:31.129030 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x"] Oct 30 22:34:31.129134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:31.129098 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-mfx7x"] Oct 30 22:34:31.234685 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:31.234651 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl"] Oct 30 22:34:31.243753 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:31.243727 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-bg6pl"] Oct 30 22:34:31.349564 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:31.349532 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp"] Oct 30 22:34:31.354758 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:31.354731 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-5rxnp"] Oct 30 22:34:32.947228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:32.925193 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" path="/var/lib/kubelet/pods/3eb179ec-ed21-46d3-88e4-45185d9e22d7/volumes" Oct 30 22:34:33.110231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:33.109993 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" path="/var/lib/kubelet/pods/5678dc0a-2ca6-41ce-852f-064f3566ce4b/volumes" Oct 30 22:34:33.114228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:34:33.110858 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" path="/var/lib/kubelet/pods/f8c2a7ce-218e-4096-a541-f016e8f53f00/volumes" Oct 30 22:34:54.411366 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:34:54.411328 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:35:01.264350 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:01.264254 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:35:02.906808 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:02.906770 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv"] Oct 30 22:35:07.907595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:07.907553 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" podUID="1137a642-65b5-4f84-95aa-2cc3fe901416" containerName="payload-processing" containerID="cri-o://5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d" gracePeriod=25 Oct 30 22:35:08.251624 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.251588 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:35:08.293752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.293720 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1137a642-65b5-4f84-95aa-2cc3fe901416-kube-api-access-6bcst\" (UniqueName: \"kubernetes.io/projected/1137a642-65b5-4f84-95aa-2cc3fe901416-kube-api-access-6bcst\") pod \"1137a642-65b5-4f84-95aa-2cc3fe901416\" (UID: \"1137a642-65b5-4f84-95aa-2cc3fe901416\") " Oct 30 22:35:08.293910 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.293770 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1137a642-65b5-4f84-95aa-2cc3fe901416-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a642-65b5-4f84-95aa-2cc3fe901416-plugins-config-volume\") pod \"1137a642-65b5-4f84-95aa-2cc3fe901416\" (UID: \"1137a642-65b5-4f84-95aa-2cc3fe901416\") " Oct 30 22:35:08.294180 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.294155 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1137a642-65b5-4f84-95aa-2cc3fe901416-plugins-config-volume" pod "1137a642-65b5-4f84-95aa-2cc3fe901416" (UID: "1137a642-65b5-4f84-95aa-2cc3fe901416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:08.295825 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.295804 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1137a642-65b5-4f84-95aa-2cc3fe901416-kube-api-access-6bcst" pod "1137a642-65b5-4f84-95aa-2cc3fe901416" (UID: "1137a642-65b5-4f84-95aa-2cc3fe901416"). InnerVolumeSpecName "kube-api-access-6bcst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:08.389802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.389761 5226 generic.go:356] "Generic (PLEG): container finished" podID="1137a642-65b5-4f84-95aa-2cc3fe901416" containerID="5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d" exitCode=0 Oct 30 22:35:08.389996 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.389808 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" Oct 30 22:35:08.389996 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.389828 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" event={"ID":"1137a642-65b5-4f84-95aa-2cc3fe901416","Type":"ContainerDied","Data":"5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d"} Oct 30 22:35:08.389996 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.389909 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv" event={"ID":"1137a642-65b5-4f84-95aa-2cc3fe901416","Type":"ContainerDied","Data":"73630eb7d692794294b46c4a61adec28c2327306e8508c1692acba0401a0593e"} Oct 30 22:35:08.389996 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.389931 5226 scope.go:122] "RemoveContainer" containerID="5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d" Oct 30 22:35:08.396127 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.396099 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bcst\" (UniqueName: \"kubernetes.io/projected/1137a642-65b5-4f84-95aa-2cc3fe901416-kube-api-access-6bcst\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:08.396279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.396131 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1137a642-65b5-4f84-95aa-2cc3fe901416-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:08.402626 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.402607 5226 scope.go:122] "RemoveContainer" containerID="5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d" Oct 30 22:35:08.403087 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:35:08.403065 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d\": container with ID starting with 5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d not found: ID does not exist" containerID="5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d" Oct 30 22:35:08.403197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.403092 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d"} err="failed to get container status \"5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d\": rpc error: code = NotFound desc = could not find container \"5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d\": container with ID starting with 5c23c160e0582d7abb451ba156074efa048ea1c113a02ba4e86f64404a0dac6d not found: ID does not exist" Oct 30 22:35:08.552460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.552388 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv"] Oct 30 22:35:08.557644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.557615 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-73735c-5fc57bf7ff-dh9xv"] Oct 30 22:35:08.906791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:08.906710 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1137a642-65b5-4f84-95aa-2cc3fe901416" path="/var/lib/kubelet/pods/1137a642-65b5-4f84-95aa-2cc3fe901416/volumes" Oct 30 22:35:24.411347 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:35:24.411311 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:35:51.668972 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.668814 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4"] Oct 30 22:35:51.670554 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670520 5226 cpu_manager.go:414] "containerMap: removing container" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" Oct 30 22:35:51.670554 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670551 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670581 5226 cpu_manager.go:414] "containerMap: removing container" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670590 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670608 5226 cpu_manager.go:414] "containerMap: removing container" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670615 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670659 5226 cpu_manager.go:414] "containerMap: removing container" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670666 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670689 5226 cpu_manager.go:414] "containerMap: removing container" podUID="1137a642-65b5-4f84-95aa-2cc3fe901416" containerName="payload-processing" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670696 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1137a642-65b5-4f84-95aa-2cc3fe901416" containerName="payload-processing" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670861 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="5678dc0a-2ca6-41ce-852f-064f3566ce4b" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670878 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd77cda-1cff-4e39-8f54-93097fa58c1c" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670889 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c2a7ce-218e-4096-a541-f016e8f53f00" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670907 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb179ec-ed21-46d3-88e4-45185d9e22d7" containerName="istio-proxy" Oct 30 22:35:51.670867 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.670920 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="1137a642-65b5-4f84-95aa-2cc3fe901416" containerName="payload-processing" Oct 30 22:35:51.699657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.699621 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:51.722839 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.722807 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4"] Oct 30 22:35:51.763817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.763773 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d6b55f\"" Oct 30 22:35:51.764234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.764158 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d6b55f-dockercfg-pf4l5\"" Oct 30 22:35:51.803541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.803507 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee04e125-52ff-47e2-b35d-94abe86c6ed1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4\" (UID: \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:51.803729 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.803704 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9txc\" (UniqueName: \"kubernetes.io/projected/ee04e125-52ff-47e2-b35d-94abe86c6ed1-kube-api-access-x9txc\") pod \"payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4\" (UID: \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:51.905296 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.905266 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee04e125-52ff-47e2-b35d-94abe86c6ed1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4\" (UID: \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:51.905485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.905357 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9txc\" (UniqueName: \"kubernetes.io/projected/ee04e125-52ff-47e2-b35d-94abe86c6ed1-kube-api-access-x9txc\") pod \"payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4\" (UID: \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:51.905984 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.905957 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee04e125-52ff-47e2-b35d-94abe86c6ed1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4\" (UID: \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:51.963445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:51.963416 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9txc\" (UniqueName: \"kubernetes.io/projected/ee04e125-52ff-47e2-b35d-94abe86c6ed1-kube-api-access-x9txc\") pod \"payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4\" (UID: \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:52.030882 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:52.030852 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:52.403960 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:52.403873 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4"] Oct 30 22:35:52.409382 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:35:52.409352 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee04e125_52ff_47e2_b35d_94abe86c6ed1.slice/crio-b6db175344c454d86ec81f0d0be3c1133ad238a69ca9eda613edd901349f0db4 WatchSource:0}: Error finding container b6db175344c454d86ec81f0d0be3c1133ad238a69ca9eda613edd901349f0db4: Status 404 returned error can't find the container with id b6db175344c454d86ec81f0d0be3c1133ad238a69ca9eda613edd901349f0db4 Oct 30 22:35:52.718183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:52.718144 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" event={"ID":"ee04e125-52ff-47e2-b35d-94abe86c6ed1","Type":"ContainerStarted","Data":"cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c"} Oct 30 22:35:52.718645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:52.718200 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" event={"ID":"ee04e125-52ff-47e2-b35d-94abe86c6ed1","Type":"ContainerStarted","Data":"b6db175344c454d86ec81f0d0be3c1133ad238a69ca9eda613edd901349f0db4"} Oct 30 22:35:52.718645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:52.718362 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:35:52.876120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:52.876061 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" podStartSLOduration=1.8760453529999999 podStartE2EDuration="1.876045353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:52.827231085 +0000 UTC m=+2044.084572796" watchObservedRunningTime="2026-10-30 22:35:52.876045353 +0000 UTC m=+2044.133387063" Oct 30 22:35:54.411581 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:35:54.411531 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:35:55.966274 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:55.965792 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq"] Oct 30 22:35:55.980247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:55.974793 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.024500 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.024463 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fb3406-openshift-default-dockercfg-dltws\"" Oct 30 22:35:56.064308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.064272 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-token\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.064500 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.064404 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-certs\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.064500 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.064477 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-podinfo\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.064642 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.064615 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-credential-socket\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.064747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.064728 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-envoy\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.064792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.064769 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-data\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.064844 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.064792 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cebe87-7505-4ac6-af29-b423530a1ad3-istiod-ca-cert\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.064994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.064969 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-socket\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.065043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.065013 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqqh\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-kube-api-access-2dqqh\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.075194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.075164 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq"] Oct 30 22:35:56.166835 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.166799 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-socket\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.167048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.166848 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqqh\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-kube-api-access-2dqqh\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.167048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.166953 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-token\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.167048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.167008 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-certs\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.167259 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.167061 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-podinfo\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.167259 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.167119 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-credential-socket\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.167259 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.167194 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-envoy\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.167259 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.167246 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-data\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.167475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.167275 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cebe87-7505-4ac6-af29-b423530a1ad3-istiod-ca-cert\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.168343 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.168315 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cebe87-7505-4ac6-af29-b423530a1ad3-istiod-ca-cert\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.168696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.168624 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-socket\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.169616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.169372 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-certs\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.169616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.169549 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-data\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.169616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.169605 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-credential-socket\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.172116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.172090 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-envoy\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.172717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.172692 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-podinfo\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.242269 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.240981 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-token\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.250103 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.250066 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqqh\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-kube-api-access-2dqqh\") pod \"e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.302969 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.302924 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:56.761401 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.761243 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq"] Oct 30 22:35:56.764490 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:35:56.764455 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cebe87_7505_4ac6_af29_b423530a1ad3.slice/crio-aa646f7701962adfa9e34647b63000554ee27a6a731bcbaf0c16db86ec02f924 WatchSource:0}: Error finding container aa646f7701962adfa9e34647b63000554ee27a6a731bcbaf0c16db86ec02f924: Status 404 returned error can't find the container with id aa646f7701962adfa9e34647b63000554ee27a6a731bcbaf0c16db86ec02f924 Oct 30 22:35:56.767456 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.767419 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:35:56.767569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.767497 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:35:56.767569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:56.767529 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:35:57.750893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:57.750853 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" event={"ID":"50cebe87-7505-4ac6-af29-b423530a1ad3","Type":"ContainerStarted","Data":"a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be"} Oct 30 22:35:57.751328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:57.750913 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" event={"ID":"50cebe87-7505-4ac6-af29-b423530a1ad3","Type":"ContainerStarted","Data":"aa646f7701962adfa9e34647b63000554ee27a6a731bcbaf0c16db86ec02f924"} Oct 30 22:35:57.913428 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:57.913373 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podStartSLOduration=2.913358989 podStartE2EDuration="2.913358989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:57.910177409 +0000 UTC m=+2049.167519119" watchObservedRunningTime="2026-10-30 22:35:57.913358989 +0000 UTC m=+2049.170700698" Oct 30 22:35:58.303508 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:58.303471 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:58.310828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:58.310796 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:58.757261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:58.757224 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:35:58.758357 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:35:58.758341 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:36:00.399270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.399237 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf"] Oct 30 22:36:00.399713 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.399475 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" podUID="5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" containerName="maas-api" containerID="cri-o://f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c" gracePeriod=30 Oct 30 22:36:00.746560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.746532 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:36:00.810021 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.809972 5226 generic.go:356] "Generic (PLEG): container finished" podID="5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" containerID="f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c" exitCode=0 Oct 30 22:36:00.811727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.811704 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" Oct 30 22:36:00.812085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.812063 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" event={"ID":"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270","Type":"ContainerDied","Data":"f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c"} Oct 30 22:36:00.812284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.812257 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf" event={"ID":"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270","Type":"ContainerDied","Data":"62c7f595a901ab8d65813cd2126e1fa6a72a5694d7dc6da0d074930e31479282"} Oct 30 22:36:00.812588 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.812571 5226 scope.go:122] "RemoveContainer" containerID="f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c" Oct 30 22:36:00.829666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.829639 5226 scope.go:122] "RemoveContainer" containerID="f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c" Oct 30 22:36:00.838229 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:36:00.830617 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c\": container with ID starting with f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c not found: ID does not exist" containerID="f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c" Oct 30 22:36:00.838229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.830679 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c"} err="failed to get container status \"f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c\": rpc error: code = NotFound desc = could not find container \"f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c\": container with ID starting with f3f96ad94b668245ac67abf13bac2c4d68ef1b708bbc70fc6f83fb6b8a0f4f6c not found: ID does not exist" Oct 30 22:36:00.857245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.850685 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-kube-api-access-f29lk\" (UniqueName: \"kubernetes.io/projected/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-kube-api-access-f29lk\") pod \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " Oct 30 22:36:00.857245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.850843 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-openshift-service-ca\") pod \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " Oct 30 22:36:00.857245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.851020 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls\") pod \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " Oct 30 22:36:00.857245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.851163 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-metrics-certs\") pod \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\" (UID: \"5c6e8a7f-7402-45bf-bfd4-0db00d1c1270\") " Oct 30 22:36:00.864439 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.861923 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-openshift-service-ca" pod "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" (UID: "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:00.903472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.886827 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-metrics-certs" pod "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" (UID: "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:00.912042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.911406 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-kube-api-access-f29lk" pod "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" (UID: "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270"). InnerVolumeSpecName "kube-api-access-f29lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:00.922256 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.921906 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls" pod "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" (UID: "5c6e8a7f-7402-45bf-bfd4-0db00d1c1270"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:00.954102 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.953982 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:00.954102 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.954017 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:00.954102 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.954036 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f29lk\" (UniqueName: \"kubernetes.io/projected/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-kube-api-access-f29lk\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:00.954102 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:00.954055 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:01.229451 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:01.229417 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf"] Oct 30 22:36:01.235133 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:01.234947 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c3c2cf-69f897fbcd-g4wgf"] Oct 30 22:36:02.905776 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:02.905735 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" path="/var/lib/kubelet/pods/5c6e8a7f-7402-45bf-bfd4-0db00d1c1270/volumes" Oct 30 22:36:03.294369 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.294340 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27"] Oct 30 22:36:03.294902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.294890 5226 cpu_manager.go:414] "containerMap: removing container" podUID="5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" containerName="maas-api" Oct 30 22:36:03.294945 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.294903 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" containerName="maas-api" Oct 30 22:36:03.295026 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.295017 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6e8a7f-7402-45bf-bfd4-0db00d1c1270" containerName="maas-api" Oct 30 22:36:03.299995 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.299975 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.348810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.348782 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-29bbe735-serving-cert\"" Oct 30 22:36:03.400169 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.400131 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27"] Oct 30 22:36:03.400488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.400463 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmlm\" (UniqueName: \"kubernetes.io/projected/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-kube-api-access-bqmlm\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.400604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.400504 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-openshift-service-ca\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.400604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.400541 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-metrics-certs\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.400842 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.400816 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.502227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.502173 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-openshift-service-ca\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.502387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.502261 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-metrics-certs\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.502464 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.502446 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.502598 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:36:03.502577 5226 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-serving-cert: secret "maas-api-e2e-mt-29bbe735-serving-cert" not found Oct 30 22:36:03.502650 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:36:03.502640 5226 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls podName:8d8661fd-7679-4fc5-8d16-ed77a4ba8446 nodeName:}" failed. No retries permitted until 2026-10-30 22:36:04.002624034 +0000 UTC m=+2055.259965753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls") pod "maas-api-e2e-mt-29bbe735-7948869679-m7m27" (UID: "8d8661fd-7679-4fc5-8d16-ed77a4ba8446") : secret "maas-api-e2e-mt-29bbe735-serving-cert" not found Oct 30 22:36:03.502711 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.502657 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmlm\" (UniqueName: \"kubernetes.io/projected/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-kube-api-access-bqmlm\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.502952 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.502928 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-openshift-service-ca\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.504745 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.504725 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-metrics-certs\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.564452 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.564366 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmlm\" (UniqueName: \"kubernetes.io/projected/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-kube-api-access-bqmlm\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:03.727881 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:03.727852 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:36:04.013939 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:04.013908 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:04.016318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:04.016292 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls\") pod \"maas-api-e2e-mt-29bbe735-7948869679-m7m27\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:04.213714 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:04.213679 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:04.650156 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:04.650059 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27"] Oct 30 22:36:04.654655 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:36:04.654626 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8661fd_7679_4fc5_8d16_ed77a4ba8446.slice/crio-ee86254124fb9420053cf65ed8246f136f03d7bc721b608651ae8b05351e07ad WatchSource:0}: Error finding container ee86254124fb9420053cf65ed8246f136f03d7bc721b608651ae8b05351e07ad: Status 404 returned error can't find the container with id ee86254124fb9420053cf65ed8246f136f03d7bc721b608651ae8b05351e07ad Oct 30 22:36:04.850082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:04.849905 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" event={"ID":"8d8661fd-7679-4fc5-8d16-ed77a4ba8446","Type":"ContainerStarted","Data":"ee86254124fb9420053cf65ed8246f136f03d7bc721b608651ae8b05351e07ad"} Oct 30 22:36:05.858405 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:05.858363 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" event={"ID":"8d8661fd-7679-4fc5-8d16-ed77a4ba8446","Type":"ContainerStarted","Data":"0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4"} Oct 30 22:36:05.858787 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:05.858459 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:05.970851 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:05.970798 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" podStartSLOduration=1.9770127739999999 podStartE2EDuration="2.970783211s" totalImagesPullingTime="993.770437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:05.969366602 +0000 UTC m=+2057.226708313" watchObservedRunningTime="2026-10-30 22:36:05.970783211 +0000 UTC m=+2057.228124921" Oct 30 22:36:11.870750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:11.870717 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:36:19.810155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:19.810119 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:36:24.411491 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:36:24.411451 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:36:25.867728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:25.867696 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq"] Oct 30 22:36:25.868135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:25.867899 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" containerID="cri-o://185eb5c36354a4c01fb796019c15ec7d43448169b1224eb709c692597d2d6c7c" gracePeriod=30 Oct 30 22:36:26.658145 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:26.658110 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6"] Oct 30 22:36:26.658362 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:26.658331 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" containerID="cri-o://2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a" gracePeriod=30 Oct 30 22:36:26.757067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:26.757029 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs"] Oct 30 22:36:26.757273 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:26.757250 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" containerID="cri-o://1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61" gracePeriod=30 Oct 30 22:36:27.053136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:27.053097 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:27.053702 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:27.053159 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:29.475328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:29.475236 5226 patch_prober.go:29] interesting pod/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:29.475328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:29.475292 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:30.193837 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:30.193792 5226 patch_prober.go:29] interesting pod/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:30.194033 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:30.193859 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:31.040430 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.040398 5226 generic.go:356] "Generic (PLEG): container finished" podID="72337f40-32a4-4136-b4e4-4b8e49399673" containerID="185eb5c36354a4c01fb796019c15ec7d43448169b1224eb709c692597d2d6c7c" exitCode=0 Oct 30 22:36:31.040810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.040466 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" event={"ID":"72337f40-32a4-4136-b4e4-4b8e49399673","Type":"ContainerDied","Data":"185eb5c36354a4c01fb796019c15ec7d43448169b1224eb709c692597d2d6c7c"} Oct 30 22:36:31.139233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.139167 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:36:31.281468 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281441 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72337f40-32a4-4136-b4e4-4b8e49399673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72337f40-32a4-4136-b4e4-4b8e49399673-istiod-ca-cert\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.281468 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281474 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-envoy\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.281721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281517 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-data\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.281721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281567 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-credential-socket\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.281721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281614 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72337f40-32a4-4136-b4e4-4b8e49399673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72337f40-32a4-4136-b4e4-4b8e49399673-istio-podinfo\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.281887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281759 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-kube-api-access-85kbf\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-kube-api-access-85kbf\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.281887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281829 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-certs\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.281887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281850 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72337f40-32a4-4136-b4e4-4b8e49399673-istiod-ca-cert" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:31.282013 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281886 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-socket\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.282013 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281905 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-credential-socket" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.282013 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.281994 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-istio-token\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-istio-token\") pod \"72337f40-32a4-4136-b4e4-4b8e49399673\" (UID: \"72337f40-32a4-4136-b4e4-4b8e49399673\") " Oct 30 22:36:31.282183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.282101 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-certs" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.282183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.282150 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-data" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.282183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.282166 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-socket" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.282662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.282634 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72337f40-32a4-4136-b4e4-4b8e49399673-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.282783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.282665 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.282783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.282681 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.282783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.282695 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.282783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.282726 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.283932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.283907 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-envoy" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:31.284037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.283996 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72337f40-32a4-4136-b4e4-4b8e49399673-istio-podinfo" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:31.284037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.284009 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-kube-api-access-85kbf" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "kube-api-access-85kbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:31.284113 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.284067 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-istio-token" pod "72337f40-32a4-4136-b4e4-4b8e49399673" (UID: "72337f40-32a4-4136-b4e4-4b8e49399673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:31.384943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.384908 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.384943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.384945 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72337f40-32a4-4136-b4e4-4b8e49399673-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.385140 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.384960 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72337f40-32a4-4136-b4e4-4b8e49399673-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.385140 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.384976 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85kbf\" (UniqueName: \"kubernetes.io/projected/72337f40-32a4-4136-b4e4-4b8e49399673-kube-api-access-85kbf\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:31.946559 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:31.946529 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:36:32.039073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.039047 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:36:32.048636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.048609 5226 generic.go:356] "Generic (PLEG): container finished" podID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerID="2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a" exitCode=0 Oct 30 22:36:32.049016 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.048653 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" Oct 30 22:36:32.049016 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.048696 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" event={"ID":"498f6c92-51cd-4ee0-a05d-34d8fa471345","Type":"ContainerDied","Data":"2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a"} Oct 30 22:36:32.049016 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.048747 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6" event={"ID":"498f6c92-51cd-4ee0-a05d-34d8fa471345","Type":"ContainerDied","Data":"1f4c9b835eb0603fafb6d69ca36ddddd3fec9d537efc0f4c81f14d038f204a6e"} Oct 30 22:36:32.049016 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.048779 5226 scope.go:122] "RemoveContainer" containerID="2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a" Oct 30 22:36:32.050130 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.050107 5226 generic.go:356] "Generic (PLEG): container finished" podID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerID="1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61" exitCode=0 Oct 30 22:36:32.050292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.050157 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" Oct 30 22:36:32.050292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.050185 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" event={"ID":"16dfee67-2e63-498b-ad2c-efbdd34146ae","Type":"ContainerDied","Data":"1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61"} Oct 30 22:36:32.050292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.050267 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs" event={"ID":"16dfee67-2e63-498b-ad2c-efbdd34146ae","Type":"ContainerDied","Data":"959dac9a1af7eca194f2dbbba053b2553e095e850bf3984e7f6647d6b4ff42ce"} Oct 30 22:36:32.051589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.051571 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" Oct 30 22:36:32.051690 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.051624 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq" event={"ID":"72337f40-32a4-4136-b4e4-4b8e49399673","Type":"ContainerDied","Data":"bf2a9d78508d1b8557164b78dcd06581dcfd2d3a03e3e7c3ebfe2ea318bf0234"} Oct 30 22:36:32.061001 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.060983 5226 scope.go:122] "RemoveContainer" containerID="2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a" Oct 30 22:36:32.061432 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:36:32.061405 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a\": container with ID starting with 2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a not found: ID does not exist" containerID="2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a" Oct 30 22:36:32.061540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.061442 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a"} err="failed to get container status \"2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a\": rpc error: code = NotFound desc = could not find container \"2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a\": container with ID starting with 2685e43504e6734e1861e2173970774b1b0a9a80cd63468a81379a925bcc963a not found: ID does not exist" Oct 30 22:36:32.061540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.061464 5226 scope.go:122] "RemoveContainer" containerID="1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61" Oct 30 22:36:32.072869 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.072850 5226 scope.go:122] "RemoveContainer" containerID="1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61" Oct 30 22:36:32.073155 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:36:32.073133 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61\": container with ID starting with 1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61 not found: ID does not exist" containerID="1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61" Oct 30 22:36:32.073264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.073168 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61"} err="failed to get container status \"1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61\": rpc error: code = NotFound desc = could not find container \"1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61\": container with ID starting with 1b9c4b0835ffa442b465e008e26dbd9116d6c6861882e03c116cb090e472bb61 not found: ID does not exist" Oct 30 22:36:32.073264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.073192 5226 scope.go:122] "RemoveContainer" containerID="185eb5c36354a4c01fb796019c15ec7d43448169b1224eb709c692597d2d6c7c" Oct 30 22:36:32.095530 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.095508 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-token\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-token\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.095635 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.095621 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/498f6c92-51cd-4ee0-a05d-34d8fa471345-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498f6c92-51cd-4ee0-a05d-34d8fa471345-istiod-ca-cert\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.095688 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.095674 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-kube-api-access-qqzvg\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-kube-api-access-qqzvg\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.095761 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.095739 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-credential-socket\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.095823 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.095787 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-certs\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.095823 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.095815 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-socket\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.095954 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.095935 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-envoy\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.096008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.095977 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-podinfo\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.096063 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.096020 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-data\") pod \"498f6c92-51cd-4ee0-a05d-34d8fa471345\" (UID: \"498f6c92-51cd-4ee0-a05d-34d8fa471345\") " Oct 30 22:36:32.096141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.096115 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-credential-socket" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.096245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.096171 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498f6c92-51cd-4ee0-a05d-34d8fa471345-istiod-ca-cert" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:32.097088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.096533 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-data" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.097088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.096730 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.097088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.096750 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/498f6c92-51cd-4ee0-a05d-34d8fa471345-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.097088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.096766 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.097088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.096969 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-socket" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.097088 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.097021 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-certs" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.098694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.098668 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-podinfo" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:32.098694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.098684 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-envoy" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.098928 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.098909 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-token" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:32.099002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.098942 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-kube-api-access-qqzvg" pod "498f6c92-51cd-4ee0-a05d-34d8fa471345" (UID: "498f6c92-51cd-4ee0-a05d-34d8fa471345"). InnerVolumeSpecName "kube-api-access-qqzvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:32.197423 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197337 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-credential-socket\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.197423 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197387 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16dfee67-2e63-498b-ad2c-efbdd34146ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dfee67-2e63-498b-ad2c-efbdd34146ae-istiod-ca-cert\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.197656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197431 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-socket\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.197656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197476 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-data\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.197656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197619 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-credential-socket" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.197656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197646 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-token\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-token\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.197860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197739 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-socket" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.197860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197753 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-data" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.197860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197774 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-podinfo\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.197860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197795 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16dfee67-2e63-498b-ad2c-efbdd34146ae-istiod-ca-cert" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:32.198056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197918 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-certs\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.198056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197956 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-envoy\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.198056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.197981 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-kube-api-access-fb7vs\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-kube-api-access-fb7vs\") pod \"16dfee67-2e63-498b-ad2c-efbdd34146ae\" (UID: \"16dfee67-2e63-498b-ad2c-efbdd34146ae\") " Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198326 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-certs" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198629 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198650 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198666 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqzvg\" (UniqueName: \"kubernetes.io/projected/498f6c92-51cd-4ee0-a05d-34d8fa471345-kube-api-access-qqzvg\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198684 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198701 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198716 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198730 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198744 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16dfee67-2e63-498b-ad2c-efbdd34146ae-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198757 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198771 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.198821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.198785 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/498f6c92-51cd-4ee0-a05d-34d8fa471345-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.200514 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.200348 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-token" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:32.200514 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.200485 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-envoy" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:32.200723 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.200533 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-kube-api-access-fb7vs" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "kube-api-access-fb7vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:32.200864 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.200845 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-podinfo" pod "16dfee67-2e63-498b-ad2c-efbdd34146ae" (UID: "16dfee67-2e63-498b-ad2c-efbdd34146ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:32.300607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.300570 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.300607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.300599 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.300607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.300612 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb7vs\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-kube-api-access-fb7vs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.300893 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.300627 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16dfee67-2e63-498b-ad2c-efbdd34146ae-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:32.349096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.349060 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq"] Oct 30 22:36:32.354011 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.353978 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-lvzmq"] Oct 30 22:36:32.466630 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.466546 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6"] Oct 30 22:36:32.471231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.471189 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-tjcx6"] Oct 30 22:36:32.573332 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.573298 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs"] Oct 30 22:36:32.577543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.577506 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-qmmqs"] Oct 30 22:36:32.906407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.906375 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" path="/var/lib/kubelet/pods/16dfee67-2e63-498b-ad2c-efbdd34146ae/volumes" Oct 30 22:36:32.906894 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.906879 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" path="/var/lib/kubelet/pods/498f6c92-51cd-4ee0-a05d-34d8fa471345/volumes" Oct 30 22:36:32.907346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:32.907333 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" path="/var/lib/kubelet/pods/72337f40-32a4-4136-b4e4-4b8e49399673/volumes" Oct 30 22:36:49.330403 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:49.330374 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:36:49.342014 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:36:49.341990 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:36:54.411876 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:36:54.411837 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:37:07.013746 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:07.013710 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:37:11.749911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:11.749876 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x"] Oct 30 22:37:11.750352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:11.750124 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" podUID="012c0719-751f-42c0-bb56-0776dae3569e" containerName="discovery" containerID="cri-o://a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4" gracePeriod=30 Oct 30 22:37:12.050308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.050282 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:37:12.150769 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.150732 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-ca-configmap\") pod \"012c0719-751f-42c0-bb56-0776dae3569e\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " Oct 30 22:37:12.150986 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.150789 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-kube-api-access-q5glp\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-kube-api-access-q5glp\") pod \"012c0719-751f-42c0-bb56-0776dae3569e\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " Oct 30 22:37:12.150986 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.150862 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012c0719-751f-42c0-bb56-0776dae3569e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/012c0719-751f-42c0-bb56-0776dae3569e-local-certs\") pod \"012c0719-751f-42c0-bb56-0776dae3569e\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " Oct 30 22:37:12.150986 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.150915 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-istio-token\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-istio-token\") pod \"012c0719-751f-42c0-bb56-0776dae3569e\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " Oct 30 22:37:12.152305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.151276 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-ca-configmap" pod "012c0719-751f-42c0-bb56-0776dae3569e" (UID: "012c0719-751f-42c0-bb56-0776dae3569e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:12.152305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.151495 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-cacerts\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-cacerts\") pod \"012c0719-751f-42c0-bb56-0776dae3569e\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " Oct 30 22:37:12.152305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.151627 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-dns-cert\") pod \"012c0719-751f-42c0-bb56-0776dae3569e\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " Oct 30 22:37:12.152305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.151733 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-kubeconfig\") pod \"012c0719-751f-42c0-bb56-0776dae3569e\" (UID: \"012c0719-751f-42c0-bb56-0776dae3569e\") " Oct 30 22:37:12.152705 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.152495 5226 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-ca-configmap\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.154120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.154083 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-cacerts" pod "012c0719-751f-42c0-bb56-0776dae3569e" (UID: "012c0719-751f-42c0-bb56-0776dae3569e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:12.154281 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.154133 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012c0719-751f-42c0-bb56-0776dae3569e-local-certs" pod "012c0719-751f-42c0-bb56-0776dae3569e" (UID: "012c0719-751f-42c0-bb56-0776dae3569e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:12.155153 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.155127 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-kubeconfig" pod "012c0719-751f-42c0-bb56-0776dae3569e" (UID: "012c0719-751f-42c0-bb56-0776dae3569e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:12.155301 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.155189 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-kube-api-access-q5glp" pod "012c0719-751f-42c0-bb56-0776dae3569e" (UID: "012c0719-751f-42c0-bb56-0776dae3569e"). InnerVolumeSpecName "kube-api-access-q5glp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:12.155371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.155317 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-istio-token" pod "012c0719-751f-42c0-bb56-0776dae3569e" (UID: "012c0719-751f-42c0-bb56-0776dae3569e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:12.155572 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.155552 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-dns-cert" pod "012c0719-751f-42c0-bb56-0776dae3569e" (UID: "012c0719-751f-42c0-bb56-0776dae3569e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:12.253617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.253581 5226 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/012c0719-751f-42c0-bb56-0776dae3569e-local-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.253617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.253616 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.253888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.253630 5226 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-cacerts\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.253888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.253645 5226 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-csr-dns-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.253888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.253659 5226 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/012c0719-751f-42c0-bb56-0776dae3569e-istio-kubeconfig\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.253888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.253673 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5glp\" (UniqueName: \"kubernetes.io/projected/012c0719-751f-42c0-bb56-0776dae3569e-kube-api-access-q5glp\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:12.311676 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.311590 5226 generic.go:356] "Generic (PLEG): container finished" podID="012c0719-751f-42c0-bb56-0776dae3569e" containerID="a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4" exitCode=0 Oct 30 22:37:12.311676 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.311630 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" Oct 30 22:37:12.311861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.311676 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" event={"ID":"012c0719-751f-42c0-bb56-0776dae3569e","Type":"ContainerDied","Data":"a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4"} Oct 30 22:37:12.311861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.311728 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x" event={"ID":"012c0719-751f-42c0-bb56-0776dae3569e","Type":"ContainerDied","Data":"170080382b5c406a48be9ef9c828d81a1bd8039954b4a4d47bd327ba45595b95"} Oct 30 22:37:12.311861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.311751 5226 scope.go:122] "RemoveContainer" containerID="a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4" Oct 30 22:37:12.322805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.322790 5226 scope.go:122] "RemoveContainer" containerID="a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4" Oct 30 22:37:12.323058 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:37:12.323037 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4\": container with ID starting with a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4 not found: ID does not exist" containerID="a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4" Oct 30 22:37:12.323160 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.323065 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4"} err="failed to get container status \"a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4\": rpc error: code = NotFound desc = could not find container \"a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4\": container with ID starting with a80cf6e3da42d2cf9f69694ca0b936d74d3fee68ca580d676c70b91839f9d0d4 not found: ID does not exist" Oct 30 22:37:12.467304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.467269 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x"] Oct 30 22:37:12.473616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.473589 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fxm8x"] Oct 30 22:37:12.905273 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:12.905238 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012c0719-751f-42c0-bb56-0776dae3569e" path="/var/lib/kubelet/pods/012c0719-751f-42c0-bb56-0776dae3569e/volumes" Oct 30 22:37:21.044496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.044462 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26"] Oct 30 22:37:21.045231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045213 5226 cpu_manager.go:414] "containerMap: removing container" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" Oct 30 22:37:21.045307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045233 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" Oct 30 22:37:21.045307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045257 5226 cpu_manager.go:414] "containerMap: removing container" podUID="012c0719-751f-42c0-bb56-0776dae3569e" containerName="discovery" Oct 30 22:37:21.045307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045265 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012c0719-751f-42c0-bb56-0776dae3569e" containerName="discovery" Oct 30 22:37:21.045307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045282 5226 cpu_manager.go:414] "containerMap: removing container" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" Oct 30 22:37:21.045307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045291 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" Oct 30 22:37:21.045461 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045332 5226 cpu_manager.go:414] "containerMap: removing container" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" Oct 30 22:37:21.045461 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045340 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" Oct 30 22:37:21.045526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045519 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="16dfee67-2e63-498b-ad2c-efbdd34146ae" containerName="istio-proxy" Oct 30 22:37:21.045526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045537 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="012c0719-751f-42c0-bb56-0776dae3569e" containerName="discovery" Oct 30 22:37:21.045526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045548 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="498f6c92-51cd-4ee0-a05d-34d8fa471345" containerName="istio-proxy" Oct 30 22:37:21.045691 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.045560 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="72337f40-32a4-4136-b4e4-4b8e49399673" containerName="istio-proxy" Oct 30 22:37:21.050824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.050802 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.099883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.099849 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fb3406-serving-cert\"" Oct 30 22:37:21.115957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.115925 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26"] Oct 30 22:37:21.170567 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.170530 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-metrics-certs\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.170728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.170597 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9789bc21-7597-4d25-b8a2-43c22c6fe11d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.170815 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.170792 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852bp\" (UniqueName: \"kubernetes.io/projected/9789bc21-7597-4d25-b8a2-43c22c6fe11d-kube-api-access-852bp\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.170931 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.170910 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.272569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.272344 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852bp\" (UniqueName: \"kubernetes.io/projected/9789bc21-7597-4d25-b8a2-43c22c6fe11d-kube-api-access-852bp\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.272569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.272531 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.272826 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.272737 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-metrics-certs\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.272826 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.272801 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9789bc21-7597-4d25-b8a2-43c22c6fe11d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.273412 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:37:21.272950 5226 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-serving-cert: secret "maas-api-e2e-inf-a-fb3406-serving-cert" not found Oct 30 22:37:21.273412 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:37:21.273076 5226 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls podName:9789bc21-7597-4d25-b8a2-43c22c6fe11d nodeName:}" failed. No retries permitted until 2026-10-30 22:37:21.773036504 +0000 UTC m=+2133.030378199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls") pod "maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" (UID: "9789bc21-7597-4d25-b8a2-43c22c6fe11d") : secret "maas-api-e2e-inf-a-fb3406-serving-cert" not found Oct 30 22:37:21.273772 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.273750 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9789bc21-7597-4d25-b8a2-43c22c6fe11d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.276942 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.276911 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-metrics-certs\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.330807 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.330674 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-852bp\" (UniqueName: \"kubernetes.io/projected/9789bc21-7597-4d25-b8a2-43c22c6fe11d-kube-api-access-852bp\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.781646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.781599 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.784044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.784014 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls\") pod \"maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:21.964593 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:21.964557 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:22.339522 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:22.339487 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26"] Oct 30 22:37:22.344987 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:37:22.344958 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9789bc21_7597_4d25_b8a2_43c22c6fe11d.slice/crio-4ae5bf1b202333746d93ecd1f8428cb02b17fb951e624470691905bec62d452d WatchSource:0}: Error finding container 4ae5bf1b202333746d93ecd1f8428cb02b17fb951e624470691905bec62d452d: Status 404 returned error can't find the container with id 4ae5bf1b202333746d93ecd1f8428cb02b17fb951e624470691905bec62d452d Oct 30 22:37:22.379686 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:22.379651 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" event={"ID":"9789bc21-7597-4d25-b8a2-43c22c6fe11d","Type":"ContainerStarted","Data":"4ae5bf1b202333746d93ecd1f8428cb02b17fb951e624470691905bec62d452d"} Oct 30 22:37:23.389404 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.389365 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" event={"ID":"9789bc21-7597-4d25-b8a2-43c22c6fe11d","Type":"ContainerStarted","Data":"99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c"} Oct 30 22:37:23.389860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.389532 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:23.514389 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.514284 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" podStartSLOduration=2.6355648069999997 podStartE2EDuration="3.514269515s" totalImagesPullingTime="878.704708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:23.510896321 +0000 UTC m=+2134.768238028" watchObservedRunningTime="2026-10-30 22:37:23.514269515 +0000 UTC m=+2134.771611219" Oct 30 22:37:23.719611 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.719578 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74"] Oct 30 22:37:23.723958 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.723936 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.773118 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.773047 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-29bbe735-openshift-default-dockercfg-sgjmc\"" Oct 30 22:37:23.786021 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.785993 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74"] Oct 30 22:37:23.809564 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.809535 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.809730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.809608 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.809730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.809705 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslc4\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-kube-api-access-xslc4\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.809825 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.809806 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.809954 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.809926 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.810037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.810021 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.810137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.810121 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d669c932-50e6-4d88-b925-f74424639d8b-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.810234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.810190 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.810327 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.810302 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d669c932-50e6-4d88-b925-f74424639d8b-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.911895 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.911861 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912065 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.911923 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912065 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.911965 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912065 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912018 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d669c932-50e6-4d88-b925-f74424639d8b-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912065 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912044 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912081 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d669c932-50e6-4d88-b925-f74424639d8b-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912152 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912245 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912304 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslc4\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-kube-api-access-xslc4\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912306 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912351 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912557 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912637 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.912868 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.912850 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d669c932-50e6-4d88-b925-f74424639d8b-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.914461 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.914439 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.914774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.914751 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d669c932-50e6-4d88-b925-f74424639d8b-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.991225 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.991116 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:23.993807 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:23.993774 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslc4\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-kube-api-access-xslc4\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:24.037615 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:24.037518 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:24.412747 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:37:24.412654 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:37:24.474360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:24.474314 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74"] Oct 30 22:37:24.479428 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:37:24.479400 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd669c932_50e6_4d88_b925_f74424639d8b.slice/crio-cdf3a9be395f6d3c28ca1a0c8616df79f0d3723cf60fc2d4793f63c8d490f763 WatchSource:0}: Error finding container cdf3a9be395f6d3c28ca1a0c8616df79f0d3723cf60fc2d4793f63c8d490f763: Status 404 returned error can't find the container with id cdf3a9be395f6d3c28ca1a0c8616df79f0d3723cf60fc2d4793f63c8d490f763 Oct 30 22:37:24.482165 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:24.482132 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:24.482272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:24.482198 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:24.482272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:24.482243 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:25.404617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:25.404573 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" event={"ID":"d669c932-50e6-4d88-b925-f74424639d8b","Type":"ContainerStarted","Data":"5e85b7c90ebc2a4d0d077b1b4de09cb641bd9f44dc0a2a6eda3a2c6c47097efe"} Oct 30 22:37:25.404775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:25.404631 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" event={"ID":"d669c932-50e6-4d88-b925-f74424639d8b","Type":"ContainerStarted","Data":"cdf3a9be395f6d3c28ca1a0c8616df79f0d3723cf60fc2d4793f63c8d490f763"} Oct 30 22:37:25.520417 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:25.520360 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" podStartSLOduration=2.520343612 podStartE2EDuration="2.520343612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:25.517944653 +0000 UTC m=+2136.775286362" watchObservedRunningTime="2026-10-30 22:37:25.520343612 +0000 UTC m=+2136.777685324" Oct 30 22:37:26.042231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:26.038558 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:26.047227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:26.043314 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:37:26.047227 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:26.043380 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:37:27.043231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:27.041594 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:37:27.043231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:27.041651 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:37:28.041275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:28.041239 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:37:28.041561 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:28.041530 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:37:29.045745 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:29.045621 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:37:29.048334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:29.046259 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:37:29.401540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:29.401462 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:37:30.066235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:30.065430 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:37:30.066235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:30.065481 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 22:37:31.515816 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:31.515786 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:31.516343 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:31.516324 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:31.517585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:31.517562 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:37:37.184357 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.184320 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9"] Oct 30 22:37:37.184875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.184573 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" podUID="45dcc87b-99f4-4f75-851c-2bd67975600d" containerName="maas-api" containerID="cri-o://0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74" gracePeriod=30 Oct 30 22:37:37.493457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.493429 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:37:37.536926 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.536892 5226 generic.go:356] "Generic (PLEG): container finished" podID="45dcc87b-99f4-4f75-851c-2bd67975600d" containerID="0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74" exitCode=0 Oct 30 22:37:37.537073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.536957 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" Oct 30 22:37:37.537073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.536984 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" event={"ID":"45dcc87b-99f4-4f75-851c-2bd67975600d","Type":"ContainerDied","Data":"0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74"} Oct 30 22:37:37.537073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.537053 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9" event={"ID":"45dcc87b-99f4-4f75-851c-2bd67975600d","Type":"ContainerDied","Data":"f311e168ca2089ddd5f9e742773cf5d200ec037c3883ffd1c7884ad5291d3c7d"} Oct 30 22:37:37.537197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.537087 5226 scope.go:122] "RemoveContainer" containerID="0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74" Oct 30 22:37:37.552089 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.552063 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45dcc87b-99f4-4f75-851c-2bd67975600d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dcc87b-99f4-4f75-851c-2bd67975600d-openshift-service-ca\") pod \"45dcc87b-99f4-4f75-851c-2bd67975600d\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " Oct 30 22:37:37.552478 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.552454 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45dcc87b-99f4-4f75-851c-2bd67975600d-kube-api-access-x5wg6\" (UniqueName: \"kubernetes.io/projected/45dcc87b-99f4-4f75-851c-2bd67975600d-kube-api-access-x5wg6\") pod \"45dcc87b-99f4-4f75-851c-2bd67975600d\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " Oct 30 22:37:37.552595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.552554 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-metrics-certs\") pod \"45dcc87b-99f4-4f75-851c-2bd67975600d\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " Oct 30 22:37:37.552665 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.552593 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls\") pod \"45dcc87b-99f4-4f75-851c-2bd67975600d\" (UID: \"45dcc87b-99f4-4f75-851c-2bd67975600d\") " Oct 30 22:37:37.553081 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.553045 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45dcc87b-99f4-4f75-851c-2bd67975600d-openshift-service-ca" pod "45dcc87b-99f4-4f75-851c-2bd67975600d" (UID: "45dcc87b-99f4-4f75-851c-2bd67975600d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:37.553576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.553555 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45dcc87b-99f4-4f75-851c-2bd67975600d-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:37.575362 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.568384 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45dcc87b-99f4-4f75-851c-2bd67975600d-kube-api-access-x5wg6" pod "45dcc87b-99f4-4f75-851c-2bd67975600d" (UID: "45dcc87b-99f4-4f75-851c-2bd67975600d"). InnerVolumeSpecName "kube-api-access-x5wg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:37.575362 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.575300 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-metrics-certs" pod "45dcc87b-99f4-4f75-851c-2bd67975600d" (UID: "45dcc87b-99f4-4f75-851c-2bd67975600d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:37.586241 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.579637 5226 scope.go:122] "RemoveContainer" containerID="0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74" Oct 30 22:37:37.593447 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:37:37.593370 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74\": container with ID starting with 0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74 not found: ID does not exist" containerID="0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74" Oct 30 22:37:37.593447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.593410 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74"} err="failed to get container status \"0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74\": rpc error: code = NotFound desc = could not find container \"0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74\": container with ID starting with 0799d6e08ca2c3af32b3ab4e6a539b1e7de66437ad4150d4da1e43e32a969a74 not found: ID does not exist" Oct 30 22:37:37.629908 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.627466 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls" pod "45dcc87b-99f4-4f75-851c-2bd67975600d" (UID: "45dcc87b-99f4-4f75-851c-2bd67975600d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:37:37.664226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.657605 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5wg6\" (UniqueName: \"kubernetes.io/projected/45dcc87b-99f4-4f75-851c-2bd67975600d-kube-api-access-x5wg6\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:37.664226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.657647 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:37.664226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.657664 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45dcc87b-99f4-4f75-851c-2bd67975600d-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:37.976321 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.976284 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9"] Oct 30 22:37:37.990527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:37.990493 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d57980-547c9d85b5-ps9d9"] Oct 30 22:37:38.855428 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.855392 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c"] Oct 30 22:37:38.856387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.856361 5226 cpu_manager.go:414] "containerMap: removing container" podUID="45dcc87b-99f4-4f75-851c-2bd67975600d" containerName="maas-api" Oct 30 22:37:38.856387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.856387 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45dcc87b-99f4-4f75-851c-2bd67975600d" containerName="maas-api" Oct 30 22:37:38.856550 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.856522 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="45dcc87b-99f4-4f75-851c-2bd67975600d" containerName="maas-api" Oct 30 22:37:38.862486 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.862466 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.905810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.905764 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45dcc87b-99f4-4f75-851c-2bd67975600d" path="/var/lib/kubelet/pods/45dcc87b-99f4-4f75-851c-2bd67975600d/volumes" Oct 30 22:37:38.921912 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.921877 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c"] Oct 30 22:37:38.977886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.977851 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df56b81-f264-41d6-8e2e-f72094bff85b-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.978085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.977898 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.978085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.978049 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.978272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.978161 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.978272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.978234 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.978444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.978276 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.978444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.978332 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.978444 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.978417 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:38.978607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:38.978476 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2296\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-kube-api-access-s2296\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.079604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079565 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.079805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079622 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.079805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079679 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.079805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079713 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2296\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-kube-api-access-s2296\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.079805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079786 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df56b81-f264-41d6-8e2e-f72094bff85b-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079809 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079848 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079931 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.079986 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080281 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.080083 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080281 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.080143 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080281 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.080251 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.080353 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.080529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.080510 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df56b81-f264-41d6-8e2e-f72094bff85b-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.081965 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.081947 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.082382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.082365 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.136666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.136598 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.136833 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.136809 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2296\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-kube-api-access-s2296\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.176080 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.176053 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:39.610979 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.610940 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c"] Oct 30 22:37:39.615308 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:37:39.615278 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df56b81_f264_41d6_8e2e_f72094bff85b.slice/crio-94cded05861bebee5348e23f5e2ea270ab908948eb723aa37ffb4fc2ed555b7f WatchSource:0}: Error finding container 94cded05861bebee5348e23f5e2ea270ab908948eb723aa37ffb4fc2ed555b7f: Status 404 returned error can't find the container with id 94cded05861bebee5348e23f5e2ea270ab908948eb723aa37ffb4fc2ed555b7f Oct 30 22:37:39.618138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.618105 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:39.618271 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.618173 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:39.618271 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:39.618216 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:40.617233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:40.614966 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" event={"ID":"0df56b81-f264-41d6-8e2e-f72094bff85b","Type":"ContainerStarted","Data":"671d9dbfd831244626b6c23f70ba1979c61f7c695c15dde2a641b05c214f799d"} Oct 30 22:37:40.617233 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:40.615267 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" event={"ID":"0df56b81-f264-41d6-8e2e-f72094bff85b","Type":"ContainerStarted","Data":"94cded05861bebee5348e23f5e2ea270ab908948eb723aa37ffb4fc2ed555b7f"} Oct 30 22:37:40.840229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:40.829810 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" podStartSLOduration=2.829784559 podStartE2EDuration="2.829784559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:40.775307861 +0000 UTC m=+2152.032649571" watchObservedRunningTime="2026-10-30 22:37:40.829784559 +0000 UTC m=+2152.087126269" Oct 30 22:37:41.176408 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:41.176369 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:41.177834 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:41.177803 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:37:41.177981 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:41.177863 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:37:42.177028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:42.176987 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:37:42.177419 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:42.177057 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:37:43.176991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:43.176950 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:37:43.177194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:43.177019 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:37:44.176681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:44.176651 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:37:44.176863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:44.176708 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:37:45.176559 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:45.176523 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 22:37:45.176975 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:45.176591 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 22:37:46.217429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:46.217391 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:46.218176 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:46.218145 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:46.218920 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:46.218903 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:37:53.979371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:53.979331 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w"] Oct 30 22:37:53.985782 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:53.985756 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.031395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.031358 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.031561 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.031422 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.031628 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.031602 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.031696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.031655 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87628644-07c4-4cb2-9c89-37c67a769ee4-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.031744 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.031693 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.031880 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.031857 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grjrf\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-kube-api-access-grjrf\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.031919 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.031912 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.031962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.031931 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.032082 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.032062 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.088171 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.088135 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w"] Oct 30 22:37:54.133674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.133639 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.133674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.133677 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.133943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.133752 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.133943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.133824 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.133943 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.133876 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.134110 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.133991 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.134110 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.134046 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87628644-07c4-4cb2-9c89-37c67a769ee4-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.134110 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.134088 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.134377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.134119 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-credential-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.134377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.134139 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grjrf\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-kube-api-access-grjrf\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.134948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.134919 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87628644-07c4-4cb2-9c89-37c67a769ee4-istiod-ca-cert\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.135266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.135240 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-data\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.135475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.135335 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-certs\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.135475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.135370 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-socket\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.137362 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.137339 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-envoy\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.137709 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.137686 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-podinfo\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.196392 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.196368 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grjrf\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-kube-api-access-grjrf\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.196604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.196577 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-token\") pod \"e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.300962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.300891 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:54.415456 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:37:54.415414 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:37:54.740255 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.740195 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w"] Oct 30 22:37:54.749178 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:37:54.749147 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87628644_07c4_4cb2_9c89_37c67a769ee4.slice/crio-7cfa64da114a43d0f618848996e44016ac4659395fbe10c96affd32bca3659c5 WatchSource:0}: Error finding container 7cfa64da114a43d0f618848996e44016ac4659395fbe10c96affd32bca3659c5: Status 404 returned error can't find the container with id 7cfa64da114a43d0f618848996e44016ac4659395fbe10c96affd32bca3659c5 Oct 30 22:37:54.751696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.751661 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:54.751790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.751728 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:54.751790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.751755 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:37:54.817431 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:54.817401 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" event={"ID":"87628644-07c4-4cb2-9c89-37c67a769ee4","Type":"ContainerStarted","Data":"7cfa64da114a43d0f618848996e44016ac4659395fbe10c96affd32bca3659c5"} Oct 30 22:37:55.850335 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:55.850293 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" event={"ID":"87628644-07c4-4cb2-9c89-37c67a769ee4","Type":"ContainerStarted","Data":"b0768f7b3be96822090d1e75e9466ad9a10d6ec74b34d8e245fd379a151126ab"} Oct 30 22:37:56.017350 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:56.017297 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" podStartSLOduration=3.017282945 podStartE2EDuration="3.017282945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:56.015039872 +0000 UTC m=+2167.272381582" watchObservedRunningTime="2026-10-30 22:37:56.017282945 +0000 UTC m=+2167.274624655" Oct 30 22:37:56.303900 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:56.301557 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:37:56.313475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:56.313434 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:37:56.315820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:56.315785 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 22:37:56.554702 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:56.554618 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:37:57.302004 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:57.301976 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:37:57.302413 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:57.302019 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 22:37:58.301845 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:58.301812 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:37:58.302030 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:58.301870 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 22:37:59.302179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:59.302103 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:37:59.302179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:37:59.302153 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 22:38:00.302094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:00.302058 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:38:00.302262 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:00.302110 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 22:38:01.302120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:01.302076 5226 patch_prober.go:29] interesting pod/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:38:01.302380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:01.302135 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 22:38:02.307261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:02.307233 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:38:02.307670 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:02.307488 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:38:02.308395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:02.308373 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:38:08.983118 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:08.983076 5226 generic.go:356] "Generic (PLEG): container finished" podID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerID="a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be" exitCode=0 Oct 30 22:38:08.983666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:08.983155 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" event={"ID":"50cebe87-7505-4ac6-af29-b423530a1ad3","Type":"ContainerDied","Data":"a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be"} Oct 30 22:38:08.983666 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:08.983639 5226 scope.go:122] "RemoveContainer" containerID="a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be" Oct 30 22:38:08.985293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:08.985250 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:38:08.985414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:08.985348 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:38:08.985414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:08.985400 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:38:09.997312 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:09.997266 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" event={"ID":"50cebe87-7505-4ac6-af29-b423530a1ad3","Type":"ContainerStarted","Data":"c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464"} Oct 30 22:38:10.303539 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:10.303457 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:38:10.304760 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:10.304732 5226 patch_prober.go:29] interesting pod/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:38:10.304886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:10.304792 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 22:38:11.303938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:11.303903 5226 patch_prober.go:29] interesting pod/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:38:11.304471 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:11.303972 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 22:38:12.303824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:12.303791 5226 patch_prober.go:29] interesting pod/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:38:12.304002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:12.303839 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 22:38:13.303982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:13.303948 5226 patch_prober.go:29] interesting pod/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:38:13.304174 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:13.303999 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 22:38:13.758155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:13.758120 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:38:14.303431 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:14.303387 5226 patch_prober.go:29] interesting pod/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:38:14.303637 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:14.303459 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 22:38:15.303886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:15.303852 5226 patch_prober.go:29] interesting pod/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:38:15.304320 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:15.303907 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 22:38:16.307674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:16.307634 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:38:16.308776 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:16.308753 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:38:24.413292 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:38:24.413247 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:38:31.707296 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:31.707262 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s"] Oct 30 22:38:31.753255 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:31.753223 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w"] Oct 30 22:38:31.756250 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:31.756199 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl"] Oct 30 22:38:37.556738 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:37.556698 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27"] Oct 30 22:38:37.557644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:37.557607 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" podUID="8d8661fd-7679-4fc5-8d16-ed77a4ba8446" containerName="maas-api" containerID="cri-o://0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4" gracePeriod=30 Oct 30 22:38:38.065679 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.065645 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:38:38.140494 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.140394 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls\") pod \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " Oct 30 22:38:38.140494 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.140460 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-metrics-certs\") pod \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " Oct 30 22:38:38.140755 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.140514 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-kube-api-access-bqmlm\" (UniqueName: \"kubernetes.io/projected/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-kube-api-access-bqmlm\") pod \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " Oct 30 22:38:38.140755 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.140645 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-openshift-service-ca\") pod \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\" (UID: \"8d8661fd-7679-4fc5-8d16-ed77a4ba8446\") " Oct 30 22:38:38.141132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.141080 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-openshift-service-ca" pod "8d8661fd-7679-4fc5-8d16-ed77a4ba8446" (UID: "8d8661fd-7679-4fc5-8d16-ed77a4ba8446"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:38.141326 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.141282 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:38.143244 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.143177 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls" pod "8d8661fd-7679-4fc5-8d16-ed77a4ba8446" (UID: "8d8661fd-7679-4fc5-8d16-ed77a4ba8446"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:38.143663 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.143643 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-metrics-certs" pod "8d8661fd-7679-4fc5-8d16-ed77a4ba8446" (UID: "8d8661fd-7679-4fc5-8d16-ed77a4ba8446"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:38.144066 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.144048 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-kube-api-access-bqmlm" pod "8d8661fd-7679-4fc5-8d16-ed77a4ba8446" (UID: "8d8661fd-7679-4fc5-8d16-ed77a4ba8446"). InnerVolumeSpecName "kube-api-access-bqmlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:38.242527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.242447 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmlm\" (UniqueName: \"kubernetes.io/projected/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-kube-api-access-bqmlm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:38.242527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.242486 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:38.242527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.242501 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d8661fd-7679-4fc5-8d16-ed77a4ba8446-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:38.282412 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.278116 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" Oct 30 22:38:38.283328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.283290 5226 generic.go:356] "Generic (PLEG): container finished" podID="8d8661fd-7679-4fc5-8d16-ed77a4ba8446" containerID="0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4" exitCode=0 Oct 30 22:38:38.283452 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.283384 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" event={"ID":"8d8661fd-7679-4fc5-8d16-ed77a4ba8446","Type":"ContainerDied","Data":"0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4"} Oct 30 22:38:38.283452 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.283430 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27" event={"ID":"8d8661fd-7679-4fc5-8d16-ed77a4ba8446","Type":"ContainerDied","Data":"ee86254124fb9420053cf65ed8246f136f03d7bc721b608651ae8b05351e07ad"} Oct 30 22:38:38.283555 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.283457 5226 scope.go:122] "RemoveContainer" containerID="0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4" Oct 30 22:38:38.306981 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.306869 5226 scope.go:122] "RemoveContainer" containerID="0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4" Oct 30 22:38:38.307364 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:38:38.307336 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4\": container with ID starting with 0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4 not found: ID does not exist" containerID="0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4" Oct 30 22:38:38.307503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.307373 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4"} err="failed to get container status \"0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4\": rpc error: code = NotFound desc = could not find container \"0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4\": container with ID starting with 0be98bc9eb7c804e7fbd6b622eed1362f3803086900f6f231f74b8c012c064a4 not found: ID does not exist" Oct 30 22:38:38.317014 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:38:38.316990 5226 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8661fd_7679_4fc5_8d16_ed77a4ba8446.slice/crio-ee86254124fb9420053cf65ed8246f136f03d7bc721b608651ae8b05351e07ad\": RecentStats: unable to find data in memory cache]" Oct 30 22:38:38.438707 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.438677 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27"] Oct 30 22:38:38.443258 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.443231 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29bbe735-7948869679-m7m27"] Oct 30 22:38:38.906272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:38.906167 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8661fd-7679-4fc5-8d16-ed77a4ba8446" path="/var/lib/kubelet/pods/8d8661fd-7679-4fc5-8d16-ed77a4ba8446/volumes" Oct 30 22:38:45.912098 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:45.912056 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:38:46.728489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:46.728435 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" containerID="cri-o://f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220" gracePeriod=45 Oct 30 22:38:46.773654 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:46.773603 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" containerID="cri-o://2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe" gracePeriod=45 Oct 30 22:38:46.776057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:46.776024 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" containerID="cri-o://efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485" gracePeriod=45 Oct 30 22:38:47.099138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.099115 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:38:47.104438 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.104418 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:38:47.108102 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.108082 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:38:47.175476 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175440 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-tmp-dir\") pod \"3e362626-cf7a-42c5-8017-2fb7946bd12e\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " Oct 30 22:38:47.175661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175499 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-model-cache\") pod \"7458e0f5-fb73-4a8f-9554-40f302510a37\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " Oct 30 22:38:47.175661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175538 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tls-certs\") pod \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " Oct 30 22:38:47.175661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175585 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-tmp-dir\") pod \"7458e0f5-fb73-4a8f-9554-40f302510a37\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " Oct 30 22:38:47.175661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175628 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7458e0f5-fb73-4a8f-9554-40f302510a37-kube-api-access-bcwls\" (UniqueName: \"kubernetes.io/projected/7458e0f5-fb73-4a8f-9554-40f302510a37-kube-api-access-bcwls\") pod \"7458e0f5-fb73-4a8f-9554-40f302510a37\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " Oct 30 22:38:47.175892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175665 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-dshm\") pod \"7458e0f5-fb73-4a8f-9554-40f302510a37\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " Oct 30 22:38:47.175892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175722 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-tmp-dir" pod "3e362626-cf7a-42c5-8017-2fb7946bd12e" (UID: "3e362626-cf7a-42c5-8017-2fb7946bd12e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.175892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175737 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-model-cache" pod "7458e0f5-fb73-4a8f-9554-40f302510a37" (UID: "7458e0f5-fb73-4a8f-9554-40f302510a37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.175892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175777 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-home\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-home\") pod \"3e362626-cf7a-42c5-8017-2fb7946bd12e\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " Oct 30 22:38:47.175892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175829 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-home\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-home\") pod \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " Oct 30 22:38:47.176158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175954 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-home\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-home\") pod \"7458e0f5-fb73-4a8f-9554-40f302510a37\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " Oct 30 22:38:47.176158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175962 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-tmp-dir" pod "7458e0f5-fb73-4a8f-9554-40f302510a37" (UID: "7458e0f5-fb73-4a8f-9554-40f302510a37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.176158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175994 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-home" pod "3e362626-cf7a-42c5-8017-2fb7946bd12e" (UID: "3e362626-cf7a-42c5-8017-2fb7946bd12e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.176158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.175999 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-dshm\") pod \"3e362626-cf7a-42c5-8017-2fb7946bd12e\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " Oct 30 22:38:47.176158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176074 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-model-cache\") pod \"3e362626-cf7a-42c5-8017-2fb7946bd12e\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " Oct 30 22:38:47.176460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176162 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-home" pod "9253bf18-33e8-4e12-8b40-6a39bc59abbe" (UID: "9253bf18-33e8-4e12-8b40-6a39bc59abbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.176460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176319 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9253bf18-33e8-4e12-8b40-6a39bc59abbe-kube-api-access-zxmt7\" (UniqueName: \"kubernetes.io/projected/9253bf18-33e8-4e12-8b40-6a39bc59abbe-kube-api-access-zxmt7\") pod \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " Oct 30 22:38:47.176460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176344 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-home" pod "7458e0f5-fb73-4a8f-9554-40f302510a37" (UID: "7458e0f5-fb73-4a8f-9554-40f302510a37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.176460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176355 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e362626-cf7a-42c5-8017-2fb7946bd12e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e362626-cf7a-42c5-8017-2fb7946bd12e-tls-certs\") pod \"3e362626-cf7a-42c5-8017-2fb7946bd12e\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " Oct 30 22:38:47.176460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176411 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7458e0f5-fb73-4a8f-9554-40f302510a37-tls-certs\" (UniqueName: \"kubernetes.io/secret/7458e0f5-fb73-4a8f-9554-40f302510a37-tls-certs\") pod \"7458e0f5-fb73-4a8f-9554-40f302510a37\" (UID: \"7458e0f5-fb73-4a8f-9554-40f302510a37\") " Oct 30 22:38:47.176460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176438 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tmp-dir\") pod \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " Oct 30 22:38:47.176798 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176471 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-model-cache\") pod \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " Oct 30 22:38:47.176798 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176498 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-model-cache" pod "3e362626-cf7a-42c5-8017-2fb7946bd12e" (UID: "3e362626-cf7a-42c5-8017-2fb7946bd12e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.176798 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176536 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-dshm\") pod \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\" (UID: \"9253bf18-33e8-4e12-8b40-6a39bc59abbe\") " Oct 30 22:38:47.176798 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.176601 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e362626-cf7a-42c5-8017-2fb7946bd12e-kube-api-access-vc89h\" (UniqueName: \"kubernetes.io/projected/3e362626-cf7a-42c5-8017-2fb7946bd12e-kube-api-access-vc89h\") pod \"3e362626-cf7a-42c5-8017-2fb7946bd12e\" (UID: \"3e362626-cf7a-42c5-8017-2fb7946bd12e\") " Oct 30 22:38:47.177541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.177280 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tmp-dir" pod "9253bf18-33e8-4e12-8b40-6a39bc59abbe" (UID: "9253bf18-33e8-4e12-8b40-6a39bc59abbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.178590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.178562 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tls-certs" pod "9253bf18-33e8-4e12-8b40-6a39bc59abbe" (UID: "9253bf18-33e8-4e12-8b40-6a39bc59abbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:47.179070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.178789 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7458e0f5-fb73-4a8f-9554-40f302510a37-kube-api-access-bcwls" pod "7458e0f5-fb73-4a8f-9554-40f302510a37" (UID: "7458e0f5-fb73-4a8f-9554-40f302510a37"). InnerVolumeSpecName "kube-api-access-bcwls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:47.179070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.178820 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-dshm" pod "3e362626-cf7a-42c5-8017-2fb7946bd12e" (UID: "3e362626-cf7a-42c5-8017-2fb7946bd12e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.179070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.178873 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e362626-cf7a-42c5-8017-2fb7946bd12e-tls-certs" pod "3e362626-cf7a-42c5-8017-2fb7946bd12e" (UID: "3e362626-cf7a-42c5-8017-2fb7946bd12e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:47.179070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.178882 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-model-cache" pod "9253bf18-33e8-4e12-8b40-6a39bc59abbe" (UID: "9253bf18-33e8-4e12-8b40-6a39bc59abbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.179070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179057 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-dshm" pod "7458e0f5-fb73-4a8f-9554-40f302510a37" (UID: "7458e0f5-fb73-4a8f-9554-40f302510a37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.179406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179344 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9253bf18-33e8-4e12-8b40-6a39bc59abbe-kube-api-access-zxmt7" pod "9253bf18-33e8-4e12-8b40-6a39bc59abbe" (UID: "9253bf18-33e8-4e12-8b40-6a39bc59abbe"). InnerVolumeSpecName "kube-api-access-zxmt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:47.179406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179358 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179377 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179387 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179399 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179408 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcwls\" (UniqueName: \"kubernetes.io/projected/7458e0f5-fb73-4a8f-9554-40f302510a37-kube-api-access-bcwls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179416 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179425 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179433 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179441 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7458e0f5-fb73-4a8f-9554-40f302510a37-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179449 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179457 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e362626-cf7a-42c5-8017-2fb7946bd12e-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179465 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e362626-cf7a-42c5-8017-2fb7946bd12e-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179474 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179482 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.179604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.179591 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7458e0f5-fb73-4a8f-9554-40f302510a37-tls-certs" pod "7458e0f5-fb73-4a8f-9554-40f302510a37" (UID: "7458e0f5-fb73-4a8f-9554-40f302510a37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:47.180628 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.180607 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-dshm" pod "9253bf18-33e8-4e12-8b40-6a39bc59abbe" (UID: "9253bf18-33e8-4e12-8b40-6a39bc59abbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:47.181388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.181368 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e362626-cf7a-42c5-8017-2fb7946bd12e-kube-api-access-vc89h" pod "3e362626-cf7a-42c5-8017-2fb7946bd12e" (UID: "3e362626-cf7a-42c5-8017-2fb7946bd12e"). InnerVolumeSpecName "kube-api-access-vc89h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:47.280863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.280773 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxmt7\" (UniqueName: \"kubernetes.io/projected/9253bf18-33e8-4e12-8b40-6a39bc59abbe-kube-api-access-zxmt7\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.280863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.280799 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7458e0f5-fb73-4a8f-9554-40f302510a37-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.280863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.280808 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9253bf18-33e8-4e12-8b40-6a39bc59abbe-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.280863 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.280817 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc89h\" (UniqueName: \"kubernetes.io/projected/3e362626-cf7a-42c5-8017-2fb7946bd12e-kube-api-access-vc89h\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:47.473492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.473454 5226 generic.go:356] "Generic (PLEG): container finished" podID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerID="efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485" exitCode=0 Oct 30 22:38:47.473923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.473724 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" event={"ID":"7458e0f5-fb73-4a8f-9554-40f302510a37","Type":"ContainerDied","Data":"efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485"} Oct 30 22:38:47.487371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.487342 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" event={"ID":"7458e0f5-fb73-4a8f-9554-40f302510a37","Type":"ContainerDied","Data":"2ef7b5dca02c6f8e8b990176f89719e3e95d1097f7c1a85bf58e9b887a8d48b7"} Oct 30 22:38:47.487878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.487852 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" event={"ID":"3e362626-cf7a-42c5-8017-2fb7946bd12e","Type":"ContainerDied","Data":"f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220"} Oct 30 22:38:47.488035 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.487171 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" Oct 30 22:38:47.493280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.487586 5226 scope.go:122] "RemoveContainer" containerID="efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485" Oct 30 22:38:47.493280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.473811 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl" Oct 30 22:38:47.505707 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.487101 5226 generic.go:356] "Generic (PLEG): container finished" podID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerID="f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220" exitCode=0 Oct 30 22:38:47.505707 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.503372 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" event={"ID":"3e362626-cf7a-42c5-8017-2fb7946bd12e","Type":"ContainerDied","Data":"a8342bcc5eea2410a40387dcf73adb53c205e42af8e05f354f9b890fa4e1180c"} Oct 30 22:38:47.510265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.507170 5226 generic.go:356] "Generic (PLEG): container finished" podID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerID="2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe" exitCode=0 Oct 30 22:38:47.510265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.507272 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" event={"ID":"9253bf18-33e8-4e12-8b40-6a39bc59abbe","Type":"ContainerDied","Data":"2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe"} Oct 30 22:38:47.510265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.507307 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" event={"ID":"9253bf18-33e8-4e12-8b40-6a39bc59abbe","Type":"ContainerDied","Data":"9ace0002c11c90e1bbb09cd6262c9cc8387e9884dcf41ba25fb04d4292ad5f28"} Oct 30 22:38:47.510265 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.507382 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w" Oct 30 22:38:47.543602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.542975 5226 scope.go:122] "RemoveContainer" containerID="efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485" Oct 30 22:38:47.543743 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:38:47.543682 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485\": container with ID starting with efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485 not found: ID does not exist" containerID="efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485" Oct 30 22:38:47.543743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.543721 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485"} err="failed to get container status \"efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485\": rpc error: code = NotFound desc = could not find container \"efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485\": container with ID starting with efa4209f0f3a8333d6f7b38bb6a37f9c1ba36edb53dd9c1b941482cd9f187485 not found: ID does not exist" Oct 30 22:38:47.543890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.543748 5226 scope.go:122] "RemoveContainer" containerID="f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220" Oct 30 22:38:47.560302 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.559983 5226 scope.go:122] "RemoveContainer" containerID="f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220" Oct 30 22:38:47.560860 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:38:47.560516 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220\": container with ID starting with f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220 not found: ID does not exist" containerID="f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220" Oct 30 22:38:47.560860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.560553 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220"} err="failed to get container status \"f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220\": rpc error: code = NotFound desc = could not find container \"f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220\": container with ID starting with f43952de05f7da3f70f6cd42950c6b568d25bf21b1ee62a00e8568e7125ea220 not found: ID does not exist" Oct 30 22:38:47.560860 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.560588 5226 scope.go:122] "RemoveContainer" containerID="2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe" Oct 30 22:38:47.595806 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.595784 5226 scope.go:122] "RemoveContainer" containerID="2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe" Oct 30 22:38:47.596175 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:38:47.596143 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe\": container with ID starting with 2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe not found: ID does not exist" containerID="2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe" Oct 30 22:38:47.596693 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.596200 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe"} err="failed to get container status \"2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe\": rpc error: code = NotFound desc = could not find container \"2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe\": container with ID starting with 2d87a6aae80fb873c2c5b608d2226a3800a6bcd720345ebb4bb445370029cdbe not found: ID does not exist" Oct 30 22:38:47.752251 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.751850 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s"] Oct 30 22:38:47.805278 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.784984 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s"] Oct 30 22:38:47.871725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.871684 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl"] Oct 30 22:38:47.879658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.879623 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/simulator-w5-f80a0e-kserve-5655fffb58-qqmbl"] Oct 30 22:38:47.986680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.986629 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w"] Oct 30 22:38:47.991275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:47.991246 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f80a0e/premium-w5-f80a0e-kserve-7887f6bc67-6527w"] Oct 30 22:38:48.052523 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.052481 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f80a0e/distinct-w5-f80a0e-kserve-7959786b54-m5p9s" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 22:38:48.915120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.915081 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" path="/var/lib/kubelet/pods/3e362626-cf7a-42c5-8017-2fb7946bd12e/volumes" Oct 30 22:38:48.915870 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.915846 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" path="/var/lib/kubelet/pods/7458e0f5-fb73-4a8f-9554-40f302510a37/volumes" Oct 30 22:38:48.916496 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.916472 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" path="/var/lib/kubelet/pods/9253bf18-33e8-4e12-8b40-6a39bc59abbe/volumes" Oct 30 22:38:48.951304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951274 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646"] Oct 30 22:38:48.951836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951821 5226 cpu_manager.go:414] "containerMap: removing container" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" Oct 30 22:38:48.951836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951836 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" Oct 30 22:38:48.951950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951854 5226 cpu_manager.go:414] "containerMap: removing container" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" Oct 30 22:38:48.951950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951859 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" Oct 30 22:38:48.951950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951868 5226 cpu_manager.go:414] "containerMap: removing container" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" Oct 30 22:38:48.951950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951872 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" Oct 30 22:38:48.951950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951881 5226 cpu_manager.go:414] "containerMap: removing container" podUID="8d8661fd-7679-4fc5-8d16-ed77a4ba8446" containerName="maas-api" Oct 30 22:38:48.951950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951886 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8661fd-7679-4fc5-8d16-ed77a4ba8446" containerName="maas-api" Oct 30 22:38:48.952152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.951996 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="7458e0f5-fb73-4a8f-9554-40f302510a37" containerName="main" Oct 30 22:38:48.952152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.952008 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e362626-cf7a-42c5-8017-2fb7946bd12e" containerName="main" Oct 30 22:38:48.952152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.952018 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="9253bf18-33e8-4e12-8b40-6a39bc59abbe" containerName="main" Oct 30 22:38:48.952152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.952029 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8661fd-7679-4fc5-8d16-ed77a4ba8446" containerName="maas-api" Oct 30 22:38:48.958310 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.958279 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:48.978766 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.976366 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9b8\" (UniqueName: \"kubernetes.io/projected/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-kube-api-access-9l9b8\") pod \"payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646\" (UID: \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:48.978766 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:48.976499 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646\" (UID: \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:49.010765 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.010690 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-afb98c\"" Oct 30 22:38:49.014019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.013996 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646"] Oct 30 22:38:49.053397 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.053358 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-afb98c-dockercfg-7hldl\"" Oct 30 22:38:49.143240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.129721 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9b8\" (UniqueName: \"kubernetes.io/projected/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-kube-api-access-9l9b8\") pod \"payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646\" (UID: \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:49.143240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.129917 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646\" (UID: \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:49.179238 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.174531 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646\" (UID: \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:49.191990 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.190760 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw"] Oct 30 22:38:49.196727 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.196693 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw"] Oct 30 22:38:49.196938 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.196812 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:49.233190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.233087 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9b8\" (UniqueName: \"kubernetes.io/projected/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-kube-api-access-9l9b8\") pod \"payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646\" (UID: \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:49.276633 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.276600 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:49.333537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.332705 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvl7c\" (UniqueName: \"kubernetes.io/projected/09534bf8-e3e6-420e-8172-631254b050cd-kube-api-access-cvl7c\") pod \"payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw\" (UID: \"09534bf8-e3e6-420e-8172-631254b050cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:49.333537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.332852 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09534bf8-e3e6-420e-8172-631254b050cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw\" (UID: \"09534bf8-e3e6-420e-8172-631254b050cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:49.433859 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.433776 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvl7c\" (UniqueName: \"kubernetes.io/projected/09534bf8-e3e6-420e-8172-631254b050cd-kube-api-access-cvl7c\") pod \"payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw\" (UID: \"09534bf8-e3e6-420e-8172-631254b050cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:49.434017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.433880 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09534bf8-e3e6-420e-8172-631254b050cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw\" (UID: \"09534bf8-e3e6-420e-8172-631254b050cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:49.434634 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.434614 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09534bf8-e3e6-420e-8172-631254b050cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw\" (UID: \"09534bf8-e3e6-420e-8172-631254b050cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:49.496903 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.496870 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvl7c\" (UniqueName: \"kubernetes.io/projected/09534bf8-e3e6-420e-8172-631254b050cd-kube-api-access-cvl7c\") pod \"payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw\" (UID: \"09534bf8-e3e6-420e-8172-631254b050cd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:49.517656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.517628 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:49.654610 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.654580 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646"] Oct 30 22:38:49.660432 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:38:49.660402 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3eebb59_0f23_4b2d_b862_1a1909cc4e57.slice/crio-d7d9bdfa46159e5f9409abc42c2a22da4b24f0f08f249be58dc71340132ee14c WatchSource:0}: Error finding container d7d9bdfa46159e5f9409abc42c2a22da4b24f0f08f249be58dc71340132ee14c: Status 404 returned error can't find the container with id d7d9bdfa46159e5f9409abc42c2a22da4b24f0f08f249be58dc71340132ee14c Oct 30 22:38:49.897417 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:49.897381 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw"] Oct 30 22:38:49.903050 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:38:49.903019 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09534bf8_e3e6_420e_8172_631254b050cd.slice/crio-10ed827468839e85b6113fca7e3c3bac7da82628c5e70d13fdb0b3eb2c4d6185 WatchSource:0}: Error finding container 10ed827468839e85b6113fca7e3c3bac7da82628c5e70d13fdb0b3eb2c4d6185: Status 404 returned error can't find the container with id 10ed827468839e85b6113fca7e3c3bac7da82628c5e70d13fdb0b3eb2c4d6185 Oct 30 22:38:50.533473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:50.533434 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" event={"ID":"e3eebb59-0f23-4b2d-b862-1a1909cc4e57","Type":"ContainerStarted","Data":"747c23cd3ed57b2466f2487b2133b4cda99a03609825f2f369a23934f1509a12"} Oct 30 22:38:50.533924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:50.533607 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:38:50.533924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:50.533643 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" event={"ID":"e3eebb59-0f23-4b2d-b862-1a1909cc4e57","Type":"ContainerStarted","Data":"d7d9bdfa46159e5f9409abc42c2a22da4b24f0f08f249be58dc71340132ee14c"} Oct 30 22:38:50.535075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:50.535052 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" event={"ID":"09534bf8-e3e6-420e-8172-631254b050cd","Type":"ContainerStarted","Data":"1b168d7bb26c93eadb8cd234997f0589cb8a615951107d7ed52bfe19ca65f059"} Oct 30 22:38:50.535197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:50.535084 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" event={"ID":"09534bf8-e3e6-420e-8172-631254b050cd","Type":"ContainerStarted","Data":"10ed827468839e85b6113fca7e3c3bac7da82628c5e70d13fdb0b3eb2c4d6185"} Oct 30 22:38:50.535197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:50.535185 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:38:50.644406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:50.644344 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" podStartSLOduration=2.644330508 podStartE2EDuration="2.644330508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:50.641402958 +0000 UTC m=+2221.898744671" watchObservedRunningTime="2026-10-30 22:38:50.644330508 +0000 UTC m=+2221.901672217" Oct 30 22:38:50.751454 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:38:50.751401 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" podStartSLOduration=1.7513879079999999 podStartE2EDuration="1.751387908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:50.749319475 +0000 UTC m=+2222.006661184" watchObservedRunningTime="2026-10-30 22:38:50.751387908 +0000 UTC m=+2222.008729617" Oct 30 22:38:54.415145 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:38:54.415102 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:39:01.543726 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:01.543692 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:39:01.544131 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:01.543771 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:39:08.739874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:08.739838 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp"] Oct 30 22:39:08.743783 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:08.743750 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7"] Oct 30 22:39:23.762159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:23.762117 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" containerID="cri-o://abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a" gracePeriod=45 Oct 30 22:39:23.764511 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:23.764482 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" containerID="cri-o://6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e" gracePeriod=45 Oct 30 22:39:24.038047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.038007 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:39:24.063105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.063077 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:39:24.180358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180322 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tmp-dir\") pod \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " Oct 30 22:39:24.180547 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180398 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-dshm\") pod \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " Oct 30 22:39:24.180547 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180481 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-model-cache\") pod \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " Oct 30 22:39:24.180700 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180567 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-home\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-home\") pod \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " Oct 30 22:39:24.180700 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180657 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b741662d-6efd-40f1-88f9-4fc558cd61c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/b741662d-6efd-40f1-88f9-4fc558cd61c6-tls-certs\") pod \"b741662d-6efd-40f1-88f9-4fc558cd61c6\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " Oct 30 22:39:24.180817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180709 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tls-certs\" (UniqueName: \"kubernetes.io/secret/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tls-certs\") pod \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " Oct 30 22:39:24.180817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180785 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-model-cache" pod "02c9879f-87e6-4a5b-af84-5fdaaf41ae54" (UID: "02c9879f-87e6-4a5b-af84-5fdaaf41ae54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.180932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180818 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-kube-api-access-8fmrs\" (UniqueName: \"kubernetes.io/projected/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-kube-api-access-8fmrs\") pod \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\" (UID: \"02c9879f-87e6-4a5b-af84-5fdaaf41ae54\") " Oct 30 22:39:24.180932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180870 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-dshm\") pod \"b741662d-6efd-40f1-88f9-4fc558cd61c6\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " Oct 30 22:39:24.181045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180952 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-tmp-dir\") pod \"b741662d-6efd-40f1-88f9-4fc558cd61c6\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " Oct 30 22:39:24.181045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.181004 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-model-cache\") pod \"b741662d-6efd-40f1-88f9-4fc558cd61c6\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " Oct 30 22:39:24.181045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.181040 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-home\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-home\") pod \"b741662d-6efd-40f1-88f9-4fc558cd61c6\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " Oct 30 22:39:24.181219 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.181074 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b741662d-6efd-40f1-88f9-4fc558cd61c6-kube-api-access-9b5c2\" (UniqueName: \"kubernetes.io/projected/b741662d-6efd-40f1-88f9-4fc558cd61c6-kube-api-access-9b5c2\") pod \"b741662d-6efd-40f1-88f9-4fc558cd61c6\" (UID: \"b741662d-6efd-40f1-88f9-4fc558cd61c6\") " Oct 30 22:39:24.181361 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.181330 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-tmp-dir" pod "b741662d-6efd-40f1-88f9-4fc558cd61c6" (UID: "b741662d-6efd-40f1-88f9-4fc558cd61c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.181879 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.181852 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.181979 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.181882 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.182047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.182032 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-home" pod "02c9879f-87e6-4a5b-af84-5fdaaf41ae54" (UID: "02c9879f-87e6-4a5b-af84-5fdaaf41ae54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.182300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.182272 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-model-cache" pod "b741662d-6efd-40f1-88f9-4fc558cd61c6" (UID: "b741662d-6efd-40f1-88f9-4fc558cd61c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.182490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.182464 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-home" pod "b741662d-6efd-40f1-88f9-4fc558cd61c6" (UID: "b741662d-6efd-40f1-88f9-4fc558cd61c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.182738 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.180639 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tmp-dir" pod "02c9879f-87e6-4a5b-af84-5fdaaf41ae54" (UID: "02c9879f-87e6-4a5b-af84-5fdaaf41ae54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.183886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.183861 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-dshm" pod "02c9879f-87e6-4a5b-af84-5fdaaf41ae54" (UID: "02c9879f-87e6-4a5b-af84-5fdaaf41ae54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.183982 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.183890 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tls-certs" pod "02c9879f-87e6-4a5b-af84-5fdaaf41ae54" (UID: "02c9879f-87e6-4a5b-af84-5fdaaf41ae54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:24.184058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.183980 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b741662d-6efd-40f1-88f9-4fc558cd61c6-kube-api-access-9b5c2" pod "b741662d-6efd-40f1-88f9-4fc558cd61c6" (UID: "b741662d-6efd-40f1-88f9-4fc558cd61c6"). InnerVolumeSpecName "kube-api-access-9b5c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:24.184308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.184188 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-kube-api-access-8fmrs" pod "02c9879f-87e6-4a5b-af84-5fdaaf41ae54" (UID: "02c9879f-87e6-4a5b-af84-5fdaaf41ae54"). InnerVolumeSpecName "kube-api-access-8fmrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:39:24.185300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.185273 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b741662d-6efd-40f1-88f9-4fc558cd61c6-tls-certs" pod "b741662d-6efd-40f1-88f9-4fc558cd61c6" (UID: "b741662d-6efd-40f1-88f9-4fc558cd61c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:39:24.185859 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.185839 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-dshm" pod "b741662d-6efd-40f1-88f9-4fc558cd61c6" (UID: "b741662d-6efd-40f1-88f9-4fc558cd61c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:39:24.287078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.286982 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fmrs\" (UniqueName: \"kubernetes.io/projected/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-kube-api-access-8fmrs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287026 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287047 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287066 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b741662d-6efd-40f1-88f9-4fc558cd61c6-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287080 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b5c2\" (UniqueName: \"kubernetes.io/projected/b741662d-6efd-40f1-88f9-4fc558cd61c6-kube-api-access-9b5c2\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287120 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287141 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287154 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287168 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b741662d-6efd-40f1-88f9-4fc558cd61c6-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.287465 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.287180 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02c9879f-87e6-4a5b-af84-5fdaaf41ae54-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:39:24.413287 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:39:24.413252 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:39:24.791748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.791716 5226 generic.go:356] "Generic (PLEG): container finished" podID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerID="abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a" exitCode=0 Oct 30 22:39:24.792118 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.791780 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" Oct 30 22:39:24.792118 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.791798 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" event={"ID":"b741662d-6efd-40f1-88f9-4fc558cd61c6","Type":"ContainerDied","Data":"abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a"} Oct 30 22:39:24.792118 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.791854 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp" event={"ID":"b741662d-6efd-40f1-88f9-4fc558cd61c6","Type":"ContainerDied","Data":"dac04d469b2417c1c673a50eaccfd9bf42986804a11357e77887e14f67de62f7"} Oct 30 22:39:24.792118 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.791878 5226 scope.go:122] "RemoveContainer" containerID="abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a" Oct 30 22:39:24.793355 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.793333 5226 generic.go:356] "Generic (PLEG): container finished" podID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerID="6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e" exitCode=0 Oct 30 22:39:24.793453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.793372 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" event={"ID":"02c9879f-87e6-4a5b-af84-5fdaaf41ae54","Type":"ContainerDied","Data":"6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e"} Oct 30 22:39:24.793453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.793391 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" Oct 30 22:39:24.793453 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.793398 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7" event={"ID":"02c9879f-87e6-4a5b-af84-5fdaaf41ae54","Type":"ContainerDied","Data":"d1fa1ed6551189bf1a5f134eec676a76c2c6336a67e5b5e6e3c40a7f825ee5a3"} Oct 30 22:39:24.802817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.802802 5226 scope.go:122] "RemoveContainer" containerID="abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a" Oct 30 22:39:24.803125 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:39:24.803101 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a\": container with ID starting with abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a not found: ID does not exist" containerID="abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a" Oct 30 22:39:24.803178 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.803139 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a"} err="failed to get container status \"abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a\": rpc error: code = NotFound desc = could not find container \"abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a\": container with ID starting with abc4c4cc09774999e5883309019e3ac9d0b97aa0ead6fe05f9693434666d1a7a not found: ID does not exist" Oct 30 22:39:24.803178 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.803164 5226 scope.go:122] "RemoveContainer" containerID="6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e" Oct 30 22:39:24.812503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.812484 5226 scope.go:122] "RemoveContainer" containerID="6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e" Oct 30 22:39:24.812789 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:39:24.812769 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e\": container with ID starting with 6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e not found: ID does not exist" containerID="6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e" Oct 30 22:39:24.812835 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.812797 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e"} err="failed to get container status \"6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e\": rpc error: code = NotFound desc = could not find container \"6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e\": container with ID starting with 6133aed85e5794e8bbd79c91d34eb20cd5bf27e8262fb8bd85c78665751a763e not found: ID does not exist" Oct 30 22:39:24.946349 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.946320 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7"] Oct 30 22:39:24.992447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:24.992414 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/simulator-w1-af30a9-kserve-5885f48f86-jllm7"] Oct 30 22:39:25.055725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:25.055640 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp"] Oct 30 22:39:25.062393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:25.062363 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-af30a9/premium-w1-af30a9-kserve-5cb777ddbc-2h2lp"] Oct 30 22:39:26.905795 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:26.905751 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" path="/var/lib/kubelet/pods/02c9879f-87e6-4a5b-af84-5fdaaf41ae54/volumes" Oct 30 22:39:26.906480 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:26.906459 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" path="/var/lib/kubelet/pods/b741662d-6efd-40f1-88f9-4fc558cd61c6/volumes" Oct 30 22:39:54.412258 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:39:54.412190 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:39:56.522489 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:39:56.522451 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb"] Oct 30 22:40:01.523445 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:01.523407 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" podUID="7376dc43-4a28-47ef-8905-9fdb0100c2dc" containerName="payload-pre-processing" containerID="cri-o://839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf" gracePeriod=25 Oct 30 22:40:07.219798 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:07.219760 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:40:07.654415 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:07.654300 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c"] Oct 30 22:40:07.654646 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:07.654613 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" containerID="cri-o://671d9dbfd831244626b6c23f70ba1979c61f7c695c15dde2a641b05c214f799d" gracePeriod=30 Oct 30 22:40:07.659316 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:07.659287 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w"] Oct 30 22:40:07.659550 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:07.659521 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" containerID="cri-o://b0768f7b3be96822090d1e75e9466ad9a10d6ec74b34d8e245fd379a151126ab" gracePeriod=30 Oct 30 22:40:07.669234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:07.669189 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74"] Oct 30 22:40:07.669471 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:07.669438 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" containerID="cri-o://5e85b7c90ebc2a4d0d077b1b4de09cb641bd9f44dc0a2a6eda3a2c6c47097efe" gracePeriod=30 Oct 30 22:40:08.953889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:08.953851 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" podUID="7376dc43-4a28-47ef-8905-9fdb0100c2dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:40:12.597365 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:12.597329 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx"] Oct 30 22:40:12.597800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:12.597540 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" podUID="808ad9a0-6994-4616-8b45-e28b1ef83e4e" containerName="maas-api" containerID="cri-o://045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383" gracePeriod=30 Oct 30 22:40:13.197604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.197470 5226 generic.go:356] "Generic (PLEG): container finished" podID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerID="b0768f7b3be96822090d1e75e9466ad9a10d6ec74b34d8e245fd379a151126ab" exitCode=0 Oct 30 22:40:13.197604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.197550 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" event={"ID":"87628644-07c4-4cb2-9c89-37c67a769ee4","Type":"ContainerDied","Data":"b0768f7b3be96822090d1e75e9466ad9a10d6ec74b34d8e245fd379a151126ab"} Oct 30 22:40:13.198978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.198743 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:40:13.201037 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.201014 5226 generic.go:356] "Generic (PLEG): container finished" podID="d669c932-50e6-4d88-b925-f74424639d8b" containerID="5e85b7c90ebc2a4d0d077b1b4de09cb641bd9f44dc0a2a6eda3a2c6c47097efe" exitCode=0 Oct 30 22:40:13.201292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.201197 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" event={"ID":"d669c932-50e6-4d88-b925-f74424639d8b","Type":"ContainerDied","Data":"5e85b7c90ebc2a4d0d077b1b4de09cb641bd9f44dc0a2a6eda3a2c6c47097efe"} Oct 30 22:40:13.201962 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.201728 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:40:13.204668 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.204467 5226 generic.go:356] "Generic (PLEG): container finished" podID="808ad9a0-6994-4616-8b45-e28b1ef83e4e" containerID="045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383" exitCode=0 Oct 30 22:40:13.204668 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.204512 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" Oct 30 22:40:13.204668 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.204646 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" event={"ID":"808ad9a0-6994-4616-8b45-e28b1ef83e4e","Type":"ContainerDied","Data":"045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383"} Oct 30 22:40:13.204921 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.204700 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx" event={"ID":"808ad9a0-6994-4616-8b45-e28b1ef83e4e","Type":"ContainerDied","Data":"ef0552dc513348e5ce4e097f40374123a732c35f1c251849e28c62ba257cfa5e"} Oct 30 22:40:13.204921 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.204730 5226 scope.go:122] "RemoveContainer" containerID="045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383" Oct 30 22:40:13.207630 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.207347 5226 generic.go:356] "Generic (PLEG): container finished" podID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerID="671d9dbfd831244626b6c23f70ba1979c61f7c695c15dde2a641b05c214f799d" exitCode=0 Oct 30 22:40:13.207630 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.207422 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" event={"ID":"0df56b81-f264-41d6-8e2e-f72094bff85b","Type":"ContainerDied","Data":"671d9dbfd831244626b6c23f70ba1979c61f7c695c15dde2a641b05c214f799d"} Oct 30 22:40:13.207630 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.207499 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c" Oct 30 22:40:13.219844 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.219828 5226 scope.go:122] "RemoveContainer" containerID="045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383" Oct 30 22:40:13.221362 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:40:13.221257 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383\": container with ID starting with 045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383 not found: ID does not exist" containerID="045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383" Oct 30 22:40:13.221362 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.221286 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383"} err="failed to get container status \"045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383\": rpc error: code = NotFound desc = could not find container \"045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383\": container with ID starting with 045c794709a2dc4fe84b64ec057ae713a81573bb0144576f2ec07b5e7e622383 not found: ID does not exist" Oct 30 22:40:13.221362 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.221303 5226 scope.go:122] "RemoveContainer" containerID="671d9dbfd831244626b6c23f70ba1979c61f7c695c15dde2a641b05c214f799d" Oct 30 22:40:13.294125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.294099 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:40:13.299887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.299868 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:40:13.316940 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.316917 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-socket\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.317044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.316949 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-kube-api-access-xslc4\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-kube-api-access-xslc4\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.317044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.316968 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-istio-token\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-istio-token\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.317044 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.316996 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d669c932-50e6-4d88-b925-f74424639d8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d669c932-50e6-4d88-b925-f74424639d8b-istiod-ca-cert\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.317216 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317138 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-data\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.317216 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317168 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-socket" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.317216 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317178 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-kube-api-access-s2296\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-kube-api-access-s2296\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.317381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317230 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/808ad9a0-6994-4616-8b45-e28b1ef83e4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808ad9a0-6994-4616-8b45-e28b1ef83e4e-openshift-service-ca\") pod \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " Oct 30 22:40:13.317381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317279 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-envoy\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.317381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317307 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df56b81-f264-41d6-8e2e-f72094bff85b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df56b81-f264-41d6-8e2e-f72094bff85b-istiod-ca-cert\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.317381 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317363 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-data\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.317589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317401 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-certs\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.317589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317403 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-data" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.317589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317485 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-credential-socket\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.317589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317517 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d669c932-50e6-4d88-b925-f74424639d8b-istiod-ca-cert" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:13.317589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317522 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-credential-socket\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.317852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317595 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d669c932-50e6-4d88-b925-f74424639d8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d669c932-50e6-4d88-b925-f74424639d8b-istio-podinfo\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.317852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317682 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-envoy\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.317852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317716 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-podinfo\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.317852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317743 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-credential-socket" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.317852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317749 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df56b81-f264-41d6-8e2e-f72094bff85b-istiod-ca-cert" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:13.317852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317778 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-token\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-token\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.317852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317808 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-socket\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.317852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317839 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-certs\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.318345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317870 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-certs\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.318345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317906 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-socket\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.318345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317939 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-podinfo\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.318345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317970 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-kube-api-access-grjrf\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-kube-api-access-grjrf\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.318345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318008 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls\") pod \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " Oct 30 22:40:13.318345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318054 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-metrics-certs\") pod \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " Oct 30 22:40:13.318345 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318099 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-credential-socket\") pod \"d669c932-50e6-4d88-b925-f74424639d8b\" (UID: \"d669c932-50e6-4d88-b925-f74424639d8b\") " Oct 30 22:40:13.318762 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317869 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808ad9a0-6994-4616-8b45-e28b1ef83e4e-openshift-service-ca" pod "808ad9a0-6994-4616-8b45-e28b1ef83e4e" (UID: "808ad9a0-6994-4616-8b45-e28b1ef83e4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:13.318836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.317968 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-data" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.318836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318017 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-certs" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.318836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318160 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-socket" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.318836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318179 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-credential-socket" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.318836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318409 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-credential-socket" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.318836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318411 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-socket" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.318836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318584 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-certs" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.318836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.318740 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-certs" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.319630 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.319603 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-kube-api-access-xslc4" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "kube-api-access-xslc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.319730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.319704 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-istio-token" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.319805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.319714 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-envoy" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.319805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.319773 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/808ad9a0-6994-4616-8b45-e28b1ef83e4e-kube-api-access-f9nhm\" (UniqueName: \"kubernetes.io/projected/808ad9a0-6994-4616-8b45-e28b1ef83e4e-kube-api-access-f9nhm\") pod \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\" (UID: \"808ad9a0-6994-4616-8b45-e28b1ef83e4e\") " Oct 30 22:40:13.319921 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.319824 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-token\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-token\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.319921 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.319889 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-envoy\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.320034 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.319929 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-data\") pod \"0df56b81-f264-41d6-8e2e-f72094bff85b\" (UID: \"0df56b81-f264-41d6-8e2e-f72094bff85b\") " Oct 30 22:40:13.320034 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.319959 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87628644-07c4-4cb2-9c89-37c67a769ee4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87628644-07c4-4cb2-9c89-37c67a769ee4-istiod-ca-cert\") pod \"87628644-07c4-4cb2-9c89-37c67a769ee4\" (UID: \"87628644-07c4-4cb2-9c89-37c67a769ee4\") " Oct 30 22:40:13.320546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320512 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-kube-api-access-s2296" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "kube-api-access-s2296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.320659 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320625 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-podinfo" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:13.320733 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320670 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320733 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320693 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320733 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320707 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320733 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320720 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320735 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320751 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320763 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320777 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320790 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xslc4\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-kube-api-access-xslc4\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320803 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d669c932-50e6-4d88-b925-f74424639d8b-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320818 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d669c932-50e6-4d88-b925-f74424639d8b-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320831 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320844 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2296\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-kube-api-access-s2296\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320859 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/808ad9a0-6994-4616-8b45-e28b1ef83e4e-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320871 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d669c932-50e6-4d88-b925-f74424639d8b-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320884 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0df56b81-f264-41d6-8e2e-f72094bff85b-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320897 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.320964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.320910 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.321800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.321178 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls" pod "808ad9a0-6994-4616-8b45-e28b1ef83e4e" (UID: "808ad9a0-6994-4616-8b45-e28b1ef83e4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:13.321800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.321243 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-envoy" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.321800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.321492 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-data" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.321800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.321713 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-metrics-certs" pod "808ad9a0-6994-4616-8b45-e28b1ef83e4e" (UID: "808ad9a0-6994-4616-8b45-e28b1ef83e4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:13.322017 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.321807 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87628644-07c4-4cb2-9c89-37c67a769ee4-istiod-ca-cert" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:13.322235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.322176 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-kube-api-access-grjrf" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "kube-api-access-grjrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.322388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.322367 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-podinfo" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:13.322858 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.322828 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-token" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.323078 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.323055 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d669c932-50e6-4d88-b925-f74424639d8b-istio-podinfo" pod "d669c932-50e6-4d88-b925-f74424639d8b" (UID: "d669c932-50e6-4d88-b925-f74424639d8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:13.323477 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.323455 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-token" pod "0df56b81-f264-41d6-8e2e-f72094bff85b" (UID: "0df56b81-f264-41d6-8e2e-f72094bff85b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.323582 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.323564 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808ad9a0-6994-4616-8b45-e28b1ef83e4e-kube-api-access-f9nhm" pod "808ad9a0-6994-4616-8b45-e28b1ef83e4e" (UID: "808ad9a0-6994-4616-8b45-e28b1ef83e4e"). InnerVolumeSpecName "kube-api-access-f9nhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:13.323766 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.323748 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-envoy" pod "87628644-07c4-4cb2-9c89-37c67a769ee4" (UID: "87628644-07c4-4cb2-9c89-37c67a769ee4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:13.422354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422282 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422306 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422315 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422326 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422335 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grjrf\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-kube-api-access-grjrf\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422345 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422357 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/808ad9a0-6994-4616-8b45-e28b1ef83e4e-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422366 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9nhm\" (UniqueName: \"kubernetes.io/projected/808ad9a0-6994-4616-8b45-e28b1ef83e4e-kube-api-access-f9nhm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422374 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422383 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87628644-07c4-4cb2-9c89-37c67a769ee4-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422391 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0df56b81-f264-41d6-8e2e-f72094bff85b-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422398 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87628644-07c4-4cb2-9c89-37c67a769ee4-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.422681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.422406 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d669c932-50e6-4d88-b925-f74424639d8b-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:13.934270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.926734 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c"] Oct 30 22:40:13.934270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:13.930277 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-sj28c"] Oct 30 22:40:14.021246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.021197 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx"] Oct 30 22:40:14.025129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.025103 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-f80a0e-6f89d4cbf8-9jncx"] Oct 30 22:40:14.217358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.217263 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" event={"ID":"87628644-07c4-4cb2-9c89-37c67a769ee4","Type":"ContainerDied","Data":"7cfa64da114a43d0f618848996e44016ac4659395fbe10c96affd32bca3659c5"} Oct 30 22:40:14.217358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.217326 5226 scope.go:122] "RemoveContainer" containerID="b0768f7b3be96822090d1e75e9466ad9a10d6ec74b34d8e245fd379a151126ab" Oct 30 22:40:14.217358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.217275 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w" Oct 30 22:40:14.219372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.219056 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" Oct 30 22:40:14.219372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.219049 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74" event={"ID":"d669c932-50e6-4d88-b925-f74424639d8b","Type":"ContainerDied","Data":"cdf3a9be395f6d3c28ca1a0c8616df79f0d3723cf60fc2d4793f63c8d490f763"} Oct 30 22:40:14.229309 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.229291 5226 scope.go:122] "RemoveContainer" containerID="5e85b7c90ebc2a4d0d077b1b4de09cb641bd9f44dc0a2a6eda3a2c6c47097efe" Oct 30 22:40:14.375766 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.375731 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74"] Oct 30 22:40:14.380661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.380630 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-rqm74"] Oct 30 22:40:14.494542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.494463 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w"] Oct 30 22:40:14.496987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.496962 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-29bbe735-openshift-default-cb78c776f-jhf8w"] Oct 30 22:40:14.907261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.907175 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" path="/var/lib/kubelet/pods/0df56b81-f264-41d6-8e2e-f72094bff85b/volumes" Oct 30 22:40:14.907660 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.907645 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="808ad9a0-6994-4616-8b45-e28b1ef83e4e" path="/var/lib/kubelet/pods/808ad9a0-6994-4616-8b45-e28b1ef83e4e/volumes" Oct 30 22:40:14.908085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.908072 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" path="/var/lib/kubelet/pods/87628644-07c4-4cb2-9c89-37c67a769ee4/volumes" Oct 30 22:40:14.908897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:14.908881 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d669c932-50e6-4d88-b925-f74424639d8b" path="/var/lib/kubelet/pods/d669c932-50e6-4d88-b925-f74424639d8b/volumes" Oct 30 22:40:18.953815 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:18.953772 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" podUID="7376dc43-4a28-47ef-8905-9fdb0100c2dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:40:21.783969 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:21.783907 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:40:21.857079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:21.857051 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7376dc43-4a28-47ef-8905-9fdb0100c2dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376dc43-4a28-47ef-8905-9fdb0100c2dc-plugins-config-volume\") pod \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\" (UID: \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\") " Oct 30 22:40:21.857263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:21.857134 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7376dc43-4a28-47ef-8905-9fdb0100c2dc-kube-api-access-8nkbs\" (UniqueName: \"kubernetes.io/projected/7376dc43-4a28-47ef-8905-9fdb0100c2dc-kube-api-access-8nkbs\") pod \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\" (UID: \"7376dc43-4a28-47ef-8905-9fdb0100c2dc\") " Oct 30 22:40:21.857457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:21.857434 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7376dc43-4a28-47ef-8905-9fdb0100c2dc-plugins-config-volume" pod "7376dc43-4a28-47ef-8905-9fdb0100c2dc" (UID: "7376dc43-4a28-47ef-8905-9fdb0100c2dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:21.857715 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:21.857692 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7376dc43-4a28-47ef-8905-9fdb0100c2dc-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:21.859167 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:21.859144 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7376dc43-4a28-47ef-8905-9fdb0100c2dc-kube-api-access-8nkbs" pod "7376dc43-4a28-47ef-8905-9fdb0100c2dc" (UID: "7376dc43-4a28-47ef-8905-9fdb0100c2dc"). InnerVolumeSpecName "kube-api-access-8nkbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:21.959168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:21.959133 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nkbs\" (UniqueName: \"kubernetes.io/projected/7376dc43-4a28-47ef-8905-9fdb0100c2dc-kube-api-access-8nkbs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:22.287657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.287624 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7"] Oct 30 22:40:22.287896 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.287868 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" containerID="cri-o://699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59" gracePeriod=30 Oct 30 22:40:22.310630 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.310590 5226 generic.go:356] "Generic (PLEG): container finished" podID="7376dc43-4a28-47ef-8905-9fdb0100c2dc" containerID="839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf" exitCode=0 Oct 30 22:40:22.310784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.310663 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" event={"ID":"7376dc43-4a28-47ef-8905-9fdb0100c2dc","Type":"ContainerDied","Data":"839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf"} Oct 30 22:40:22.310784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.310714 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" event={"ID":"7376dc43-4a28-47ef-8905-9fdb0100c2dc","Type":"ContainerDied","Data":"be96b580a87d2dce02ac11b95accbfc46df7759ccfdd28ead8eb2c28a749af95"} Oct 30 22:40:22.310784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.310745 5226 scope.go:122] "RemoveContainer" containerID="839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf" Oct 30 22:40:22.310971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.310908 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb" Oct 30 22:40:22.325581 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.325553 5226 scope.go:122] "RemoveContainer" containerID="839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf" Oct 30 22:40:22.325860 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:40:22.325837 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf\": container with ID starting with 839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf not found: ID does not exist" containerID="839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf" Oct 30 22:40:22.325939 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.325871 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf"} err="failed to get container status \"839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf\": rpc error: code = NotFound desc = could not find container \"839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf\": container with ID starting with 839294f012e2f75ae00faea2db47c3401415be123dc7818fb4d18071c19db4cf not found: ID does not exist" Oct 30 22:40:22.531450 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.531410 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb"] Oct 30 22:40:22.536132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.536103 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d2d4c3-5cb54fb9c9-vr6tb"] Oct 30 22:40:22.907003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:22.906971 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7376dc43-4a28-47ef-8905-9fdb0100c2dc" path="/var/lib/kubelet/pods/7376dc43-4a28-47ef-8905-9fdb0100c2dc/volumes" Oct 30 22:40:24.412106 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:40:24.412067 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:40:25.059692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:25.059661 5226 patch_prober.go:29] interesting pod/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:40:25.059848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:25.059719 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:40:27.550734 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.550707 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:40:27.644856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.644778 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-certs\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.644856 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.644841 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-token\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-token\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.645051 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.644884 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-data\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.645051 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.644942 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-envoy\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.645051 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.644980 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istiod-ca-cert\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.645254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645039 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-certs" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.645254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645084 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-socket\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.645254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645141 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-kube-api-access-hbn2h\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.645254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645171 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-credential-socket\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.645254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645241 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-podinfo\") pod \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\" (UID: \"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4\") " Oct 30 22:40:27.645541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645292 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-data" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.645541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645428 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istiod-ca-cert" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:27.645541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645432 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-socket" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.645707 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.645572 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-credential-socket" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.646067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.646005 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.646067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.646031 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.646067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.646047 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.646067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.646062 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.646402 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.646076 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.647318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.647290 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-envoy" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:27.647440 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.647418 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-kube-api-access-hbn2h" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "kube-api-access-hbn2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:27.647559 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.647542 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-token" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:27.647696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.647678 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-podinfo" pod "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" (UID: "ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:27.747230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.747179 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.747230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.747220 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.747230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.747236 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbn2h\" (UniqueName: \"kubernetes.io/projected/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-kube-api-access-hbn2h\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:27.747447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:27.747248 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:28.350199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.350161 5226 generic.go:356] "Generic (PLEG): container finished" podID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerID="699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59" exitCode=0 Oct 30 22:40:28.350422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.350239 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" Oct 30 22:40:28.350422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.350250 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" event={"ID":"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4","Type":"ContainerDied","Data":"699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59"} Oct 30 22:40:28.350422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.350303 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7" event={"ID":"ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4","Type":"ContainerDied","Data":"eef1cfcd304c8dbeb13cf7170c852dfd7f1dae7218878ca763285ef5556cccf2"} Oct 30 22:40:28.350422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.350326 5226 scope.go:122] "RemoveContainer" containerID="699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59" Oct 30 22:40:28.360980 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.360953 5226 scope.go:122] "RemoveContainer" containerID="699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59" Oct 30 22:40:28.361269 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:40:28.361250 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59\": container with ID starting with 699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59 not found: ID does not exist" containerID="699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59" Oct 30 22:40:28.361319 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.361278 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59"} err="failed to get container status \"699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59\": rpc error: code = NotFound desc = could not find container \"699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59\": container with ID starting with 699039b72d560fabf262fcafbe468754b6ed9756be0cfa3dfd0832f0d09e6a59 not found: ID does not exist" Oct 30 22:40:28.554042 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.553996 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7"] Oct 30 22:40:28.561079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.561052 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f80a0e-openshift-default-59d68db9c5-q88n7"] Oct 30 22:40:28.905987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:28.905952 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" path="/var/lib/kubelet/pods/ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4/volumes" Oct 30 22:40:51.262562 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:51.262525 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r"] Oct 30 22:40:51.263658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:51.263477 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" containerID="cri-o://f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d" gracePeriod=30 Oct 30 22:40:54.090124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:54.090082 5226 patch_prober.go:29] interesting pod/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:40:54.090588 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:54.090158 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:40:54.411583 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:40:54.411499 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:40:56.528448 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.528419 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:40:56.576737 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.576670 5226 generic.go:356] "Generic (PLEG): container finished" podID="113dec97-8335-482f-8a29-bd6bb0c04894" containerID="f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d" exitCode=0 Oct 30 22:40:56.576737 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.576722 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" Oct 30 22:40:56.576909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.576756 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" event={"ID":"113dec97-8335-482f-8a29-bd6bb0c04894","Type":"ContainerDied","Data":"f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d"} Oct 30 22:40:56.576909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.576836 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r" event={"ID":"113dec97-8335-482f-8a29-bd6bb0c04894","Type":"ContainerDied","Data":"2065a94b80acc494709f20b84fa44c57d4bedcddb0582709da5bcc9dd05cb515"} Oct 30 22:40:56.576909 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.576861 5226 scope.go:122] "RemoveContainer" containerID="f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d" Oct 30 22:40:56.586354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.586337 5226 scope.go:122] "RemoveContainer" containerID="f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d" Oct 30 22:40:56.586631 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:40:56.586612 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d\": container with ID starting with f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d not found: ID does not exist" containerID="f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d" Oct 30 22:40:56.586680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.586640 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d"} err="failed to get container status \"f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d\": rpc error: code = NotFound desc = could not find container \"f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d\": container with ID starting with f29933e89d7c189f4e0a2762ad0a925c511fbccf23ade3c9ed90c8a1accbd48d not found: ID does not exist" Oct 30 22:40:56.677467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677436 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/113dec97-8335-482f-8a29-bd6bb0c04894-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113dec97-8335-482f-8a29-bd6bb0c04894-istio-podinfo\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.677613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677487 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-certs\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.677613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677505 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-kube-api-access-ftqwf\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-kube-api-access-ftqwf\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.677613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677557 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-istio-token\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-istio-token\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.677613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677579 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-socket\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.677613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677605 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-credential-socket\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.677902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677670 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-envoy\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.677902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677698 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-data\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.677902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677730 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-certs" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.677902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677801 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113dec97-8335-482f-8a29-bd6bb0c04894-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113dec97-8335-482f-8a29-bd6bb0c04894-istiod-ca-cert\") pod \"113dec97-8335-482f-8a29-bd6bb0c04894\" (UID: \"113dec97-8335-482f-8a29-bd6bb0c04894\") " Oct 30 22:40:56.678112 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.677935 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-socket" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.678170 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.678107 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-data" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.678289 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.678267 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-credential-socket" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.678397 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.678360 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113dec97-8335-482f-8a29-bd6bb0c04894-istiod-ca-cert" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:56.678527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.678507 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.678592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.678533 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113dec97-8335-482f-8a29-bd6bb0c04894-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.678592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.678549 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.678592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.678564 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.678592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.678578 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.679778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.679757 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-kube-api-access-ftqwf" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "kube-api-access-ftqwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:56.679970 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.679945 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-envoy" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:40:56.680036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.679965 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/113dec97-8335-482f-8a29-bd6bb0c04894-istio-podinfo" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:40:56.680036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.679987 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-istio-token" pod "113dec97-8335-482f-8a29-bd6bb0c04894" (UID: "113dec97-8335-482f-8a29-bd6bb0c04894"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:56.780337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.780301 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftqwf\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-kube-api-access-ftqwf\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.780337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.780333 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113dec97-8335-482f-8a29-bd6bb0c04894-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.780337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.780342 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113dec97-8335-482f-8a29-bd6bb0c04894-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.780565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.780350 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113dec97-8335-482f-8a29-bd6bb0c04894-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:40:56.992354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.992318 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r"] Oct 30 22:40:56.996645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:56.996615 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-af30a9-openshift-default-cbc4d6b59-fb64r"] Oct 30 22:40:58.905083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:40:58.905037 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" path="/var/lib/kubelet/pods/113dec97-8335-482f-8a29-bd6bb0c04894/volumes" Oct 30 22:41:02.673780 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.673748 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k"] Oct 30 22:41:02.674324 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674307 5226 cpu_manager.go:414] "containerMap: removing container" podUID="808ad9a0-6994-4616-8b45-e28b1ef83e4e" containerName="maas-api" Oct 30 22:41:02.674385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674326 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="808ad9a0-6994-4616-8b45-e28b1ef83e4e" containerName="maas-api" Oct 30 22:41:02.674385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674343 5226 cpu_manager.go:414] "containerMap: removing container" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" Oct 30 22:41:02.674385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674349 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" Oct 30 22:41:02.674385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674358 5226 cpu_manager.go:414] "containerMap: removing container" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" Oct 30 22:41:02.674385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674364 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" Oct 30 22:41:02.674385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674371 5226 cpu_manager.go:414] "containerMap: removing container" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" Oct 30 22:41:02.674385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674375 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674404 5226 cpu_manager.go:414] "containerMap: removing container" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674409 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674420 5226 cpu_manager.go:414] "containerMap: removing container" podUID="7376dc43-4a28-47ef-8905-9fdb0100c2dc" containerName="payload-pre-processing" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674425 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7376dc43-4a28-47ef-8905-9fdb0100c2dc" containerName="payload-pre-processing" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674432 5226 cpu_manager.go:414] "containerMap: removing container" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674436 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674446 5226 cpu_manager.go:414] "containerMap: removing container" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674451 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674468 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674475 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674556 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="d669c932-50e6-4d88-b925-f74424639d8b" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674566 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="7376dc43-4a28-47ef-8905-9fdb0100c2dc" containerName="payload-pre-processing" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674572 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c9879f-87e6-4a5b-af84-5fdaaf41ae54" containerName="main" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674578 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="113dec97-8335-482f-8a29-bd6bb0c04894" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674587 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3ed2bd-990d-4800-bdbd-90ac8f2d3ed4" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674594 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="808ad9a0-6994-4616-8b45-e28b1ef83e4e" containerName="maas-api" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674600 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="87628644-07c4-4cb2-9c89-37c67a769ee4" containerName="istio-proxy" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674608 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="b741662d-6efd-40f1-88f9-4fc558cd61c6" containerName="main" Oct 30 22:41:02.674605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.674613 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df56b81-f264-41d6-8e2e-f72094bff85b" containerName="istio-proxy" Oct 30 22:41:02.679615 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.679591 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.752818 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.752780 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tmp-dir\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.752998 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.752951 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-model-cache\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.752998 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.752984 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-dshm\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.753138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.753098 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsjw\" (UniqueName: \"kubernetes.io/projected/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-kube-api-access-gmsjw\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.753138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.753125 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tls-certs\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.753315 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.753144 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-home\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.775436 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.775399 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d6b55f\"/\"test-model-d6b55f-kserve-self-signed-certs\"" Oct 30 22:41:02.775625 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.775451 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d6b55f\"/\"openshift-service-ca.crt\"" Oct 30 22:41:02.775625 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.775464 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d6b55f\"/\"kube-root-ca.crt\"" Oct 30 22:41:02.775625 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.775456 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d6b55f\"/\"default-dockercfg-csqsv\"" Oct 30 22:41:02.779928 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.779906 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k"] Oct 30 22:41:02.853892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.853855 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tmp-dir\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.854125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.853982 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-model-cache\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.854125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.854007 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-dshm\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.854125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.854069 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsjw\" (UniqueName: \"kubernetes.io/projected/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-kube-api-access-gmsjw\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.854125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.854097 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tls-certs\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.854125 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.854122 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-home\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.854517 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.854491 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tmp-dir\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.854804 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.854778 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-model-cache\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.854905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.854842 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-home\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.857020 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.856995 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-dshm\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.857173 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.857155 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tls-certs\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.912535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.912502 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsjw\" (UniqueName: \"kubernetes.io/projected/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-kube-api-access-gmsjw\") pod \"test-model-d6b55f-kserve-5c774bfb59-sbg2k\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:02.993043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:02.992950 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:03.378116 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:03.378034 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k"] Oct 30 22:41:03.620066 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:03.620030 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" event={"ID":"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a","Type":"ContainerStarted","Data":"a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385"} Oct 30 22:41:03.620066 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:03.620073 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" event={"ID":"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a","Type":"ContainerStarted","Data":"68741722ffe1e6d39204ec756042a7f69584602f3ade4ac010f78d69faaff6a3"} Oct 30 22:41:03.730029 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:03.729975 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" podStartSLOduration=1.729961283 podStartE2EDuration="1.729961283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:03.72911616 +0000 UTC m=+2354.986457870" watchObservedRunningTime="2026-10-30 22:41:03.729961283 +0000 UTC m=+2354.987302993" Oct 30 22:41:03.993193 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:03.993117 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:11.974844 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:11.974807 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q"] Oct 30 22:41:11.978955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:11.978929 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.030011 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.029981 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9fce61-serving-cert\"" Oct 30 22:41:12.078498 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.078457 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q"] Oct 30 22:41:12.090396 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.090371 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-metrics-certs\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.090543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.090402 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jjz\" (UniqueName: \"kubernetes.io/projected/7e772e6b-7473-4cd2-b906-efeca3dc1842-kube-api-access-45jjz\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.090543 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.090441 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e772e6b-7473-4cd2-b906-efeca3dc1842-openshift-service-ca\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.090629 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.090604 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.191890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.191857 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.192100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.191931 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-metrics-certs\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.192100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.191947 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jjz\" (UniqueName: \"kubernetes.io/projected/7e772e6b-7473-4cd2-b906-efeca3dc1842-kube-api-access-45jjz\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.192100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.191985 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e772e6b-7473-4cd2-b906-efeca3dc1842-openshift-service-ca\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.192100 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:41:12.192017 5226 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-serving-cert: secret "maas-api-e2e-api-b-9fce61-serving-cert" not found Oct 30 22:41:12.192100 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:41:12.192083 5226 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls podName:7e772e6b-7473-4cd2-b906-efeca3dc1842 nodeName:}" failed. No retries permitted until 2026-10-30 22:41:12.692062538 +0000 UTC m=+2363.949404225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls") pod "maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" (UID: "7e772e6b-7473-4cd2-b906-efeca3dc1842") : secret "maas-api-e2e-api-b-9fce61-serving-cert" not found Oct 30 22:41:12.192644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.192625 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e772e6b-7473-4cd2-b906-efeca3dc1842-openshift-service-ca\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.194467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.194446 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-metrics-certs\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.251883 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.251803 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jjz\" (UniqueName: \"kubernetes.io/projected/7e772e6b-7473-4cd2-b906-efeca3dc1842-kube-api-access-45jjz\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.469877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.469842 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd"] Oct 30 22:41:12.471490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.470084 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" containerID="cri-o://4be0e230e4e8caac589fc980efc0c81c3ab541be8c7be2227fba745f3e867cd9" gracePeriod=30 Oct 30 22:41:12.697079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.697045 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.699563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.699538 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls\") pod \"maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.893416 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.893387 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:12.919966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.919931 5226 patch_prober.go:29] interesting pod/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:41:12.920098 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.920004 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:41:12.997244 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:12.993740 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:13.008376 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:13.008344 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:13.270177 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:13.270102 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q"] Oct 30 22:41:13.275871 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:41:13.275844 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e772e6b_7473_4cd2_b906_efeca3dc1842.slice/crio-062ea501cd41af9acbf6403746385efe5c9c3b48585347154b26bc61f73192d1 WatchSource:0}: Error finding container 062ea501cd41af9acbf6403746385efe5c9c3b48585347154b26bc61f73192d1: Status 404 returned error can't find the container with id 062ea501cd41af9acbf6403746385efe5c9c3b48585347154b26bc61f73192d1 Oct 30 22:41:13.277195 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:13.277176 5226 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:41:13.681242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:13.681185 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" event={"ID":"7e772e6b-7473-4cd2-b906-efeca3dc1842","Type":"ContainerStarted","Data":"062ea501cd41af9acbf6403746385efe5c9c3b48585347154b26bc61f73192d1"} Oct 30 22:41:13.693216 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:13.693171 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:41:14.689937 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:14.689904 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" event={"ID":"7e772e6b-7473-4cd2-b906-efeca3dc1842","Type":"ContainerStarted","Data":"9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9"} Oct 30 22:41:14.690438 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:14.690191 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:14.826866 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:14.826813 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" podStartSLOduration=2.879038029 podStartE2EDuration="3.82679889s" totalImagesPullingTime="947.760861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:14.824248996 +0000 UTC m=+2366.081590706" watchObservedRunningTime="2026-10-30 22:41:14.82679889 +0000 UTC m=+2366.084140628" Oct 30 22:41:17.714889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.714853 5226 generic.go:356] "Generic (PLEG): container finished" podID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerID="4be0e230e4e8caac589fc980efc0c81c3ab541be8c7be2227fba745f3e867cd9" exitCode=0 Oct 30 22:41:17.715260 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.714886 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" event={"ID":"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6","Type":"ContainerDied","Data":"4be0e230e4e8caac589fc980efc0c81c3ab541be8c7be2227fba745f3e867cd9"} Oct 30 22:41:17.737307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.737251 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:41:17.772164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772136 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-token\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-token\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772301 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772189 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-data\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772301 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772259 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-envoy\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772313 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-socket\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772358 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-credential-socket\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772469 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-podinfo\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772506 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istiod-ca-cert\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772625 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772564 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-data" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:17.772681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772636 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-socket" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:17.772739 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772667 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-certs\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772739 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772703 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-credential-socket" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:17.772849 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772759 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-kube-api-access-gnfz7\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-kube-api-access-gnfz7\") pod \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\" (UID: \"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6\") " Oct 30 22:41:17.772849 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772832 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-certs" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:17.772949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.772893 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istiod-ca-cert" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:17.773186 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.773168 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:17.773284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.773192 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:17.773284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.773222 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:17.773284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.773237 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:17.773284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.773251 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:17.774475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.774450 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-token" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:17.774822 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.774796 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-kube-api-access-gnfz7" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "kube-api-access-gnfz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:17.774917 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.774897 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-envoy" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:17.774961 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.774906 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-podinfo" pod "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" (UID: "c09f3234-84e4-4ff0-b9cf-6b9e4205cec6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:17.874478 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.874450 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfz7\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-kube-api-access-gnfz7\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:17.874478 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.874476 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:17.874647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.874485 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:17.874647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:17.874494 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:18.722387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:18.722354 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" event={"ID":"c09f3234-84e4-4ff0-b9cf-6b9e4205cec6","Type":"ContainerDied","Data":"30e51a5a7d969ab3a2c7fb847534b73a78a35f8e689c5542336ec91511849d45"} Oct 30 22:41:18.722387 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:18.722381 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd" Oct 30 22:41:18.722853 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:18.722443 5226 scope.go:122] "RemoveContainer" containerID="4be0e230e4e8caac589fc980efc0c81c3ab541be8c7be2227fba745f3e867cd9" Oct 30 22:41:18.923544 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:18.923512 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd"] Oct 30 22:41:18.969618 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:18.969584 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-sfrjd"] Oct 30 22:41:20.705254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:20.705224 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:41:20.904851 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:20.904817 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" path="/var/lib/kubelet/pods/c09f3234-84e4-4ff0-b9cf-6b9e4205cec6/volumes" Oct 30 22:41:24.412142 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:41:24.412103 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:41:25.377229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:25.368624 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq"] Oct 30 22:41:25.377229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:25.368881 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" containerID="cri-o://6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7" gracePeriod=30 Oct 30 22:41:25.395719 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:25.395682 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj"] Oct 30 22:41:25.396050 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:25.396019 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" containerID="cri-o://a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e" gracePeriod=30 Oct 30 22:41:26.702514 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:26.702480 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:41:27.590393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:27.590363 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6"] Oct 30 22:41:27.590582 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:27.590555 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" containerID="cri-o://084e2d25b42c0e5f11dff26e961e43d5f53ce0ad6d84fc9301b958173a4a802f" gracePeriod=30 Oct 30 22:41:29.578353 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:29.578314 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:41:29.578774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:29.578413 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:41:29.819604 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:29.819571 5226 patch_prober.go:29] interesting pod/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:41:29.819777 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:29.819627 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:41:30.664335 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.664263 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:41:30.700313 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.700288 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:41:30.733270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733238 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-certs\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733285 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istiod-ca-cert\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733318 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-credential-socket\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733350 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-data\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733393 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-token\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-token\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733524 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-kube-api-access-959x7\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-kube-api-access-959x7\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733559 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-podinfo\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733557 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-certs" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.733704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733594 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-socket\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733894 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733716 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-envoy\") pod \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\" (UID: \"8b61aa24-c615-44fd-a2ae-fd7f075ede95\") " Oct 30 22:41:30.733894 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733740 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-credential-socket" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.733894 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733771 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istiod-ca-cert" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:30.734064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.733963 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-socket" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.734064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.734018 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-data" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.734410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.734387 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.734526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.734413 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.734526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.734428 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.734526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.734444 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.734526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.734458 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.735810 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.735785 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-token" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.735897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.735802 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-kube-api-access-959x7" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "kube-api-access-959x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.735897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.735805 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-podinfo" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:30.735974 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.735932 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-envoy" pod "8b61aa24-c615-44fd-a2ae-fd7f075ede95" (UID: "8b61aa24-c615-44fd-a2ae-fd7f075ede95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.798977 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.798927 5226 generic.go:356] "Generic (PLEG): container finished" podID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerID="6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7" exitCode=0 Oct 30 22:41:30.799274 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.799239 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" event={"ID":"8b61aa24-c615-44fd-a2ae-fd7f075ede95","Type":"ContainerDied","Data":"6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7"} Oct 30 22:41:30.799378 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.799344 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" event={"ID":"8b61aa24-c615-44fd-a2ae-fd7f075ede95","Type":"ContainerDied","Data":"8fac4d850a2f6999705c239a07fde85344ccf2213acc52c843eaf0c623666c14"} Oct 30 22:41:30.799460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.799383 5226 scope.go:122] "RemoveContainer" containerID="6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7" Oct 30 22:41:30.800092 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.799583 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq" Oct 30 22:41:30.804052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.804020 5226 generic.go:356] "Generic (PLEG): container finished" podID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerID="a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e" exitCode=0 Oct 30 22:41:30.804162 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.804093 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" event={"ID":"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72","Type":"ContainerDied","Data":"a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e"} Oct 30 22:41:30.804162 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.804136 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" event={"ID":"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72","Type":"ContainerDied","Data":"2f9a36813f3a1ab4b82751d4a03f19b1f87da92fe03f30c1935da9fa4ad6a5e4"} Oct 30 22:41:30.804285 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.804255 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj" Oct 30 22:41:30.813145 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.813127 5226 scope.go:122] "RemoveContainer" containerID="6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7" Oct 30 22:41:30.813619 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:41:30.813597 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7\": container with ID starting with 6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7 not found: ID does not exist" containerID="6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7" Oct 30 22:41:30.813718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.813625 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7"} err="failed to get container status \"6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7\": rpc error: code = NotFound desc = could not find container \"6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7\": container with ID starting with 6a9d397138769e225976cbf793ecce04632a62c016030e262020e52c32dd86f7 not found: ID does not exist" Oct 30 22:41:30.813718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.813642 5226 scope.go:122] "RemoveContainer" containerID="a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e" Oct 30 22:41:30.822587 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.822570 5226 scope.go:122] "RemoveContainer" containerID="a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e" Oct 30 22:41:30.822845 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:41:30.822825 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e\": container with ID starting with a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e not found: ID does not exist" containerID="a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e" Oct 30 22:41:30.822906 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.822852 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e"} err="failed to get container status \"a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e\": rpc error: code = NotFound desc = could not find container \"a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e\": container with ID starting with a0c0df1ed90c50ff07d68a704884d7640d7adb9fa23ecea0c6c6ff70e1e78e9e not found: ID does not exist" Oct 30 22:41:30.835792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.835761 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istiod-ca-cert\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.835949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.835857 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-credential-socket\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.835949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.835886 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-token\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-token\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.836093 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.835973 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-kube-api-access-vpjsx\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-kube-api-access-vpjsx\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.836093 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836040 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-data\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.836093 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836069 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-socket\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.836282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836102 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-certs\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.836282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836199 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istiod-ca-cert" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:30.836282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836265 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-envoy\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.836490 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836303 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-podinfo\") pod \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\" (UID: \"90044cb5-a7fd-47ac-8db9-ed74f0b7cd72\") " Oct 30 22:41:30.836560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836508 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-data" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.836560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836534 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-credential-socket" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.836961 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836905 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-959x7\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-kube-api-access-959x7\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.837058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836971 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.837058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.836988 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.837058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.837003 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.837058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.837000 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-certs" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.837058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.837018 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.837058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.837032 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.837058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.837045 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b61aa24-c615-44fd-a2ae-fd7f075ede95-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.837466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.837185 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-socket" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.839562 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.839539 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-token" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.839674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.839576 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-kube-api-access-vpjsx" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "kube-api-access-vpjsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:30.839674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.839650 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-podinfo" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:30.840100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.840084 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-envoy" pod "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" (UID: "90044cb5-a7fd-47ac-8db9-ed74f0b7cd72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:30.928873 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.928843 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq"] Oct 30 22:41:30.934691 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.934666 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-bnpjq"] Oct 30 22:41:30.938859 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.938833 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.938978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.938862 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjsx\" (UniqueName: \"kubernetes.io/projected/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-kube-api-access-vpjsx\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.938978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.938880 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.938978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.938889 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.938978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.938898 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:30.938978 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:30.938906 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.215278 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:31.215176 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj"] Oct 30 22:41:31.219564 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:31.219531 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d2d4c3-openshift-default-6567dd864f-r7fhj"] Oct 30 22:41:32.820356 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.820326 5226 generic.go:356] "Generic (PLEG): container finished" podID="cf266f4f-e337-4e0e-a852-17467ece8400" containerID="084e2d25b42c0e5f11dff26e961e43d5f53ce0ad6d84fc9301b958173a4a802f" exitCode=0 Oct 30 22:41:32.820788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.820392 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" event={"ID":"cf266f4f-e337-4e0e-a852-17467ece8400","Type":"ContainerDied","Data":"084e2d25b42c0e5f11dff26e961e43d5f53ce0ad6d84fc9301b958173a4a802f"} Oct 30 22:41:32.846435 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.846413 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:41:32.905658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.905582 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" path="/var/lib/kubelet/pods/8b61aa24-c615-44fd-a2ae-fd7f075ede95/volumes" Oct 30 22:41:32.906367 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.906347 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" path="/var/lib/kubelet/pods/90044cb5-a7fd-47ac-8db9-ed74f0b7cd72/volumes" Oct 30 22:41:32.961485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961457 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-socket\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961513 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-istio-token\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-istio-token\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961582 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-data\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961605 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-kube-api-access-9sdw7\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-kube-api-access-9sdw7\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961627 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-certs\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961660 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf266f4f-e337-4e0e-a852-17467ece8400-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf266f4f-e337-4e0e-a852-17467ece8400-istio-podinfo\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961686 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-credential-socket\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961741 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf266f4f-e337-4e0e-a852-17467ece8400-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf266f4f-e337-4e0e-a852-17467ece8400-istiod-ca-cert\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961742 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-socket" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:32.961951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961850 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-envoy\") pod \"cf266f4f-e337-4e0e-a852-17467ece8400\" (UID: \"cf266f4f-e337-4e0e-a852-17467ece8400\") " Oct 30 22:41:32.961951 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961891 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-data" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:32.962231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.961972 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-certs" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:32.962231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.962086 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-credential-socket" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:32.962356 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.962302 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf266f4f-e337-4e0e-a852-17467ece8400-istiod-ca-cert" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:32.962475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.962436 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:32.962475 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.962461 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:32.962602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.962478 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf266f4f-e337-4e0e-a852-17467ece8400-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:32.962602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.962492 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:32.962602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.962506 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:32.964019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.963995 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-istio-token" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:32.964328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.964306 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf266f4f-e337-4e0e-a852-17467ece8400-istio-podinfo" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:41:32.964419 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.964396 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-kube-api-access-9sdw7" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "kube-api-access-9sdw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:32.964582 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:32.964549 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-envoy" pod "cf266f4f-e337-4e0e-a852-17467ece8400" (UID: "cf266f4f-e337-4e0e-a852-17467ece8400"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:33.063389 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:33.063354 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.063389 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:33.063388 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdw7\" (UniqueName: \"kubernetes.io/projected/cf266f4f-e337-4e0e-a852-17467ece8400-kube-api-access-9sdw7\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.063585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:33.063403 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf266f4f-e337-4e0e-a852-17467ece8400-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.063585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:33.063415 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf266f4f-e337-4e0e-a852-17467ece8400-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:33.828388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:33.828352 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" Oct 30 22:41:33.828388 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:33.828363 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6" event={"ID":"cf266f4f-e337-4e0e-a852-17467ece8400","Type":"ContainerDied","Data":"5c4be099d6d5bded0700b7da1a74e40ce026f135009ea2dd3e86de247e0c7aa6"} Oct 30 22:41:33.828948 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:33.828427 5226 scope.go:122] "RemoveContainer" containerID="084e2d25b42c0e5f11dff26e961e43d5f53ce0ad6d84fc9301b958173a4a802f" Oct 30 22:41:34.028820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:34.028780 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6"] Oct 30 22:41:34.032854 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:34.032829 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b13755-openshift-default-656f8468c9-624v6"] Oct 30 22:41:34.905211 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:34.905166 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" path="/var/lib/kubelet/pods/cf266f4f-e337-4e0e-a852-17467ece8400/volumes" Oct 30 22:41:38.541036 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541001 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z"] Oct 30 22:41:38.541721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541700 5226 cpu_manager.go:414] "containerMap: removing container" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" Oct 30 22:41:38.541809 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541723 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" Oct 30 22:41:38.541809 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541753 5226 cpu_manager.go:414] "containerMap: removing container" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" Oct 30 22:41:38.541809 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541761 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" Oct 30 22:41:38.541809 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541794 5226 cpu_manager.go:414] "containerMap: removing container" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" Oct 30 22:41:38.541809 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541802 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" Oct 30 22:41:38.542075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541824 5226 cpu_manager.go:414] "containerMap: removing container" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" Oct 30 22:41:38.542075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541832 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" Oct 30 22:41:38.542075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541970 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b61aa24-c615-44fd-a2ae-fd7f075ede95" containerName="istio-proxy" Oct 30 22:41:38.542075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.541987 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="90044cb5-a7fd-47ac-8db9-ed74f0b7cd72" containerName="istio-proxy" Oct 30 22:41:38.542075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.542002 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09f3234-84e4-4ff0-b9cf-6b9e4205cec6" containerName="istio-proxy" Oct 30 22:41:38.542075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.542012 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf266f4f-e337-4e0e-a852-17467ece8400" containerName="istio-proxy" Oct 30 22:41:38.546845 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.546822 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.626607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.626574 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-home\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.626607 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.626614 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6fv\" (UniqueName: \"kubernetes.io/projected/94109748-986f-4293-ad0a-8795650676e9-kube-api-access-gb6fv\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.626803 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.626711 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-tmp-dir\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.626803 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.626756 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-dshm\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.626873 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.626844 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94109748-986f-4293-ad0a-8795650676e9-tls-certs\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.626924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.626912 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-model-cache\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.641963 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.641925 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d585d2\"/\"openshift-service-ca.crt\"" Oct 30 22:41:38.642132 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.641961 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d585d2\"/\"default-dockercfg-x9pxj\"" Oct 30 22:41:38.642231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.642138 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d585d2\"/\"kube-root-ca.crt\"" Oct 30 22:41:38.642307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.642254 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d585d2\"/\"test-model-d585d2-kserve-self-signed-certs\"" Oct 30 22:41:38.645359 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.645333 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z"] Oct 30 22:41:38.728123 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728093 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94109748-986f-4293-ad0a-8795650676e9-tls-certs\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.728312 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728172 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-model-cache\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.728312 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728261 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-home\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.728312 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728282 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6fv\" (UniqueName: \"kubernetes.io/projected/94109748-986f-4293-ad0a-8795650676e9-kube-api-access-gb6fv\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.728466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728319 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-tmp-dir\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.728466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728371 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-dshm\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.728676 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728649 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-model-cache\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.728816 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728680 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-home\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.728816 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.728736 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-tmp-dir\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.730527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.730510 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-dshm\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.730812 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.730793 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94109748-986f-4293-ad0a-8795650676e9-tls-certs\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.785596 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.785563 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6fv\" (UniqueName: \"kubernetes.io/projected/94109748-986f-4293-ad0a-8795650676e9-kube-api-access-gb6fv\") pod \"test-model-d585d2-kserve-7477dfffc9-zh48z\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:38.858775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:38.858688 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:39.243967 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:39.243932 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z"] Oct 30 22:41:39.250451 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:41:39.250408 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94109748_986f_4293_ad0a_8795650676e9.slice/crio-fa90780726ccbd2f2ff4fcdab46697259296dcb28f38340c474e7a28ce4771ff WatchSource:0}: Error finding container fa90780726ccbd2f2ff4fcdab46697259296dcb28f38340c474e7a28ce4771ff: Status 404 returned error can't find the container with id fa90780726ccbd2f2ff4fcdab46697259296dcb28f38340c474e7a28ce4771ff Oct 30 22:41:39.870141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:39.870098 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" event={"ID":"94109748-986f-4293-ad0a-8795650676e9","Type":"ContainerStarted","Data":"128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0"} Oct 30 22:41:39.870598 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:39.870153 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" event={"ID":"94109748-986f-4293-ad0a-8795650676e9","Type":"ContainerStarted","Data":"fa90780726ccbd2f2ff4fcdab46697259296dcb28f38340c474e7a28ce4771ff"} Oct 30 22:41:39.979997 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:39.979942 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" podStartSLOduration=1.979926357 podStartE2EDuration="1.979926357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:39.979586275 +0000 UTC m=+2391.236928003" watchObservedRunningTime="2026-10-30 22:41:39.979926357 +0000 UTC m=+2391.237268067" Oct 30 22:41:40.859145 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:40.859110 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:43.531749 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.531713 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc"] Oct 30 22:41:43.535663 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.535641 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.600534 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.600496 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fb3406\"/\"test-model-fb3406-kserve-self-signed-certs\"" Oct 30 22:41:43.600722 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.600593 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fb3406\"/\"default-dockercfg-qht4l\"" Oct 30 22:41:43.603662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.603631 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-dshm\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.603805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.603737 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-model-cache\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.603859 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.603810 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbsr\" (UniqueName: \"kubernetes.io/projected/2f91d146-b3af-4895-87f4-011002cc075a-kube-api-access-pjbsr\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.603907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.603864 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91d146-b3af-4895-87f4-011002cc075a-tls-certs\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.603907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.603884 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-tmp-dir\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.604093 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.604069 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-home\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.631028 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.630994 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fb3406\"/\"openshift-service-ca.crt\"" Oct 30 22:41:43.631198 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.631068 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fb3406\"/\"kube-root-ca.crt\"" Oct 30 22:41:43.633861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.633834 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc"] Oct 30 22:41:43.705636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.705591 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91d146-b3af-4895-87f4-011002cc075a-tls-certs\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.705636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.705640 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-tmp-dir\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.705885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.705713 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-home\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.705885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.705805 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-dshm\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.705885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.705881 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-model-cache\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.706094 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.705959 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbsr\" (UniqueName: \"kubernetes.io/projected/2f91d146-b3af-4895-87f4-011002cc075a-kube-api-access-pjbsr\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.706570 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.706544 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-home\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.706692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.706585 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-model-cache\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.706692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.706662 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-tmp-dir\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.708648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.708620 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-dshm\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.708775 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.708758 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91d146-b3af-4895-87f4-011002cc075a-tls-certs\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.763356 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.763324 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbsr\" (UniqueName: \"kubernetes.io/projected/2f91d146-b3af-4895-87f4-011002cc075a-kube-api-access-pjbsr\") pod \"test-model-fb3406-kserve-866d67f669-bcsxc\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:43.848558 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:43.848451 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:44.232632 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:44.232598 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc"] Oct 30 22:41:44.238461 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:41:44.238434 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f91d146_b3af_4895_87f4_011002cc075a.slice/crio-99a545667989caf17596eda9375adac01079428d4ea313159c176b38b74d1022 WatchSource:0}: Error finding container 99a545667989caf17596eda9375adac01079428d4ea313159c176b38b74d1022: Status 404 returned error can't find the container with id 99a545667989caf17596eda9375adac01079428d4ea313159c176b38b74d1022 Oct 30 22:41:44.905874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:44.905837 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" event={"ID":"2f91d146-b3af-4895-87f4-011002cc075a","Type":"ContainerStarted","Data":"53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9"} Oct 30 22:41:44.906322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:44.905884 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" event={"ID":"2f91d146-b3af-4895-87f4-011002cc075a","Type":"ContainerStarted","Data":"99a545667989caf17596eda9375adac01079428d4ea313159c176b38b74d1022"} Oct 30 22:41:45.016478 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:45.016426 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" podStartSLOduration=2.016412096 podStartE2EDuration="2.016412096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:45.014283386 +0000 UTC m=+2396.271625095" watchObservedRunningTime="2026-10-30 22:41:45.016412096 +0000 UTC m=+2396.273753806" Oct 30 22:41:45.849539 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:45.849496 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:48.859089 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:48.859060 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:48.872179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:48.872152 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:48.939433 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:48.939399 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:41:49.371472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:49.371440 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:41:49.384267 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:49.384243 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:41:53.849473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:53.849429 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:53.862371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:53.862345 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:53.968347 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:41:53.968318 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:41:54.412035 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:41:54.411998 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:42:00.116747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:00.116704 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:42:03.322907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:03.322868 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h"] Oct 30 22:42:10.323302 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:10.323266 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8"] Oct 30 22:42:15.324047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:15.323996 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" podUID="80a86fe5-a6d3-4b72-869c-e983f3d693d9" containerName="payload-pre-processing" containerID="cri-o://e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5" gracePeriod=25 Oct 30 22:42:18.361681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:18.361635 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" containerID="cri-o://f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519" gracePeriod=45 Oct 30 22:42:19.008359 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.008263 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:42:19.094638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.094597 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tls-certs\" (UniqueName: \"kubernetes.io/secret/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tls-certs\") pod \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " Oct 30 22:42:19.094841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.094657 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-home\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-home\") pod \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " Oct 30 22:42:19.094841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.094708 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-dshm\") pod \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " Oct 30 22:42:19.094841 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.094758 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tmp-dir\") pod \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " Oct 30 22:42:19.095010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.094870 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-model-cache\") pod \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " Oct 30 22:42:19.095010 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.094933 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4afd7e57-39d4-4adb-8306-8587cdc2a13e-kube-api-access-p8vc8\" (UniqueName: \"kubernetes.io/projected/4afd7e57-39d4-4adb-8306-8587cdc2a13e-kube-api-access-p8vc8\") pod \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\" (UID: \"4afd7e57-39d4-4adb-8306-8587cdc2a13e\") " Oct 30 22:42:19.096850 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.096816 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tmp-dir" pod "4afd7e57-39d4-4adb-8306-8587cdc2a13e" (UID: "4afd7e57-39d4-4adb-8306-8587cdc2a13e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:19.097045 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.097019 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-model-cache" pod "4afd7e57-39d4-4adb-8306-8587cdc2a13e" (UID: "4afd7e57-39d4-4adb-8306-8587cdc2a13e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:19.097275 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.097252 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-home" pod "4afd7e57-39d4-4adb-8306-8587cdc2a13e" (UID: "4afd7e57-39d4-4adb-8306-8587cdc2a13e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:19.104975 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.104949 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-dshm" pod "4afd7e57-39d4-4adb-8306-8587cdc2a13e" (UID: "4afd7e57-39d4-4adb-8306-8587cdc2a13e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:42:19.108351 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.108323 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tls-certs" pod "4afd7e57-39d4-4adb-8306-8587cdc2a13e" (UID: "4afd7e57-39d4-4adb-8306-8587cdc2a13e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:42:19.108501 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.108361 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afd7e57-39d4-4adb-8306-8587cdc2a13e-kube-api-access-p8vc8" pod "4afd7e57-39d4-4adb-8306-8587cdc2a13e" (UID: "4afd7e57-39d4-4adb-8306-8587cdc2a13e"). InnerVolumeSpecName "kube-api-access-p8vc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:19.115245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.115191 5226 generic.go:356] "Generic (PLEG): container finished" podID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerID="f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519" exitCode=0 Oct 30 22:42:19.115354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.115331 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" Oct 30 22:42:19.115418 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.115327 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" event={"ID":"4afd7e57-39d4-4adb-8306-8587cdc2a13e","Type":"ContainerDied","Data":"f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519"} Oct 30 22:42:19.115466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.115441 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h" event={"ID":"4afd7e57-39d4-4adb-8306-8587cdc2a13e","Type":"ContainerDied","Data":"51e63f4b1ccb0fe97034c5d38156823c8ea74ddab3be3bd7db2b1cb5b8ac5dbc"} Oct 30 22:42:19.115514 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.115466 5226 scope.go:122] "RemoveContainer" containerID="f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519" Oct 30 22:42:19.127523 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.127501 5226 scope.go:122] "RemoveContainer" containerID="f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519" Oct 30 22:42:19.132793 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:42:19.132767 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519\": container with ID starting with f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519 not found: ID does not exist" containerID="f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519" Oct 30 22:42:19.132955 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.132929 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519"} err="failed to get container status \"f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519\": rpc error: code = NotFound desc = could not find container \"f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519\": container with ID starting with f6c7bcd4fba6811250b1c5256b3bd446cc90925b2a437e25efadc9e4a3898519 not found: ID does not exist" Oct 30 22:42:19.229848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.229797 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.230135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.230113 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.230410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.230306 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.230410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.230333 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.230410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.230348 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4afd7e57-39d4-4adb-8306-8587cdc2a13e-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.230410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.230374 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8vc8\" (UniqueName: \"kubernetes.io/projected/4afd7e57-39d4-4adb-8306-8587cdc2a13e-kube-api-access-p8vc8\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:19.404278 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.345276 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h"] Oct 30 22:42:19.510226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:19.510157 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-be17e7/distinct-w6-be17e7-kserve-7db5b68b5d-2qf6h"] Oct 30 22:42:20.905675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:20.905636 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" path="/var/lib/kubelet/pods/4afd7e57-39d4-4adb-8306-8587cdc2a13e/volumes" Oct 30 22:42:21.895721 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:21.895683 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" podUID="80a86fe5-a6d3-4b72-869c-e983f3d693d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:42:24.411824 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:42:24.411782 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:42:31.895303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:31.895190 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" podUID="80a86fe5-a6d3-4b72-869c-e983f3d693d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:42:35.585168 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:35.585110 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:42:35.710422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:35.710387 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a86fe5-a6d3-4b72-869c-e983f3d693d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a86fe5-a6d3-4b72-869c-e983f3d693d9-plugins-config-volume\") pod \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\" (UID: \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\") " Oct 30 22:42:35.710796 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:35.710752 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a86fe5-a6d3-4b72-869c-e983f3d693d9-kube-api-access-4kbmb\" (UniqueName: \"kubernetes.io/projected/80a86fe5-a6d3-4b72-869c-e983f3d693d9-kube-api-access-4kbmb\") pod \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\" (UID: \"80a86fe5-a6d3-4b72-869c-e983f3d693d9\") " Oct 30 22:42:35.710927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:35.710814 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a86fe5-a6d3-4b72-869c-e983f3d693d9-plugins-config-volume" pod "80a86fe5-a6d3-4b72-869c-e983f3d693d9" (UID: "80a86fe5-a6d3-4b72-869c-e983f3d693d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:42:35.711358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:35.711321 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80a86fe5-a6d3-4b72-869c-e983f3d693d9-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:35.712911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:35.712889 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a86fe5-a6d3-4b72-869c-e983f3d693d9-kube-api-access-4kbmb" pod "80a86fe5-a6d3-4b72-869c-e983f3d693d9" (UID: "80a86fe5-a6d3-4b72-869c-e983f3d693d9"). InnerVolumeSpecName "kube-api-access-4kbmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:35.812416 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:35.812384 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kbmb\" (UniqueName: \"kubernetes.io/projected/80a86fe5-a6d3-4b72-869c-e983f3d693d9-kube-api-access-4kbmb\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:42:36.217542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.217505 5226 generic.go:356] "Generic (PLEG): container finished" podID="80a86fe5-a6d3-4b72-869c-e983f3d693d9" containerID="e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5" exitCode=0 Oct 30 22:42:36.217718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.217551 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" Oct 30 22:42:36.217718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.217587 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" event={"ID":"80a86fe5-a6d3-4b72-869c-e983f3d693d9","Type":"ContainerDied","Data":"e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5"} Oct 30 22:42:36.217718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.217646 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8" event={"ID":"80a86fe5-a6d3-4b72-869c-e983f3d693d9","Type":"ContainerDied","Data":"ff88ad30caf8fda38e9f3600e1826a169775e7f9f32de78c1eff9a95590cb91a"} Oct 30 22:42:36.217718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.217668 5226 scope.go:122] "RemoveContainer" containerID="e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5" Oct 30 22:42:36.227732 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.227709 5226 scope.go:122] "RemoveContainer" containerID="e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5" Oct 30 22:42:36.228005 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:42:36.227979 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5\": container with ID starting with e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5 not found: ID does not exist" containerID="e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5" Oct 30 22:42:36.228085 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.228016 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5"} err="failed to get container status \"e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5\": rpc error: code = NotFound desc = could not find container \"e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5\": container with ID starting with e851f2005a1685a49203d1e33a0aa2cd002f27efbf4c8e5818098a3c0944a7e5 not found: ID does not exist" Oct 30 22:42:36.371770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.371740 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8"] Oct 30 22:42:36.417960 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.417929 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-be17e7-86c6ddb8dc-zcxv8"] Oct 30 22:42:36.883636 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.883605 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:42:36.905998 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:42:36.905966 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a86fe5-a6d3-4b72-869c-e983f3d693d9" path="/var/lib/kubelet/pods/80a86fe5-a6d3-4b72-869c-e983f3d693d9/volumes" Oct 30 22:42:54.414921 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:42:54.414495 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:43:16.504191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:16.504159 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4"] Oct 30 22:43:16.507057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:16.505166 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" containerID="cri-o://cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2" gracePeriod=30 Oct 30 22:43:21.777925 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.777866 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:43:21.845911 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.845876 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b313574-95b0-4a9d-80f3-37c68537b527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b313574-95b0-4a9d-80f3-37c68537b527-istio-podinfo\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.845931 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-envoy\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.845972 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b313574-95b0-4a9d-80f3-37c68537b527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b313574-95b0-4a9d-80f3-37c68537b527-istiod-ca-cert\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.845996 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-certs\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846054 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-credential-socket\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846078 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-kube-api-access-7jzml\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846179 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-istio-token\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-istio-token\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846239 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-data\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846279 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-socket\") pod \"6b313574-95b0-4a9d-80f3-37c68537b527\" (UID: \"6b313574-95b0-4a9d-80f3-37c68537b527\") " Oct 30 22:43:21.846542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846390 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-credential-socket" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.846542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846495 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b313574-95b0-4a9d-80f3-37c68537b527-istiod-ca-cert" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:43:21.846645 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846611 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-data" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.846724 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846692 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-certs" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.846777 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846702 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-socket" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.846971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846946 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.846971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846971 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.847158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846985 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b313574-95b0-4a9d-80f3-37c68537b527-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.847158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.846998 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.847158 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.847011 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.848415 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.848395 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-kube-api-access-7jzml" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "kube-api-access-7jzml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:21.848569 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.848530 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-istio-token" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:21.848652 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.848632 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b313574-95b0-4a9d-80f3-37c68537b527-istio-podinfo" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:43:21.848694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.848628 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-envoy" pod "6b313574-95b0-4a9d-80f3-37c68537b527" (UID: "6b313574-95b0-4a9d-80f3-37c68537b527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:21.947927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.947885 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.947927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.947921 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b313574-95b0-4a9d-80f3-37c68537b527-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.947927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.947935 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b313574-95b0-4a9d-80f3-37c68537b527-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:21.948257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:21.947948 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzml\" (UniqueName: \"kubernetes.io/projected/6b313574-95b0-4a9d-80f3-37c68537b527-kube-api-access-7jzml\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:22.505334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.505294 5226 generic.go:356] "Generic (PLEG): container finished" podID="6b313574-95b0-4a9d-80f3-37c68537b527" containerID="cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2" exitCode=0 Oct 30 22:43:22.505532 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.505359 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" Oct 30 22:43:22.505532 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.505374 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" event={"ID":"6b313574-95b0-4a9d-80f3-37c68537b527","Type":"ContainerDied","Data":"cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2"} Oct 30 22:43:22.505532 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.505451 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4" event={"ID":"6b313574-95b0-4a9d-80f3-37c68537b527","Type":"ContainerDied","Data":"69979b4e65e4a5637aaf54e6fbb66671c657624149beb40beb886a4e856f056f"} Oct 30 22:43:22.505532 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.505474 5226 scope.go:122] "RemoveContainer" containerID="cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2" Oct 30 22:43:22.516820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.516800 5226 scope.go:122] "RemoveContainer" containerID="cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2" Oct 30 22:43:22.517117 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:43:22.517098 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2\": container with ID starting with cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2 not found: ID does not exist" containerID="cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2" Oct 30 22:43:22.517194 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.517131 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2"} err="failed to get container status \"cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2\": rpc error: code = NotFound desc = could not find container \"cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2\": container with ID starting with cf6b6e7ccb4ceddeeb4c9c7cf615e28526db413941df68b6a96993fbcd5f22b2 not found: ID does not exist" Oct 30 22:43:22.659531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.659499 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4"] Oct 30 22:43:22.705606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.705569 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-be17e7-openshift-default-7c64556c75-csxq4"] Oct 30 22:43:22.905041 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:22.904946 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" path="/var/lib/kubelet/pods/6b313574-95b0-4a9d-80f3-37c68537b527/volumes" Oct 30 22:43:24.411022 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:43:24.410987 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:43:26.121734 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:26.121691 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:43:54.411841 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:43:54.411802 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:43:56.840072 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:43:56.840038 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:44:24.411536 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:44:24.411454 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:44:54.410836 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:44:54.410795 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:44:55.264143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:44:55.264112 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:45:00.179141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179110 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx"] Oct 30 22:45:00.179752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179737 5226 cpu_manager.go:414] "containerMap: removing container" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" Oct 30 22:45:00.179821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179757 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" Oct 30 22:45:00.179821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179771 5226 cpu_manager.go:414] "containerMap: removing container" podUID="80a86fe5-a6d3-4b72-869c-e983f3d693d9" containerName="payload-pre-processing" Oct 30 22:45:00.179821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179776 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a86fe5-a6d3-4b72-869c-e983f3d693d9" containerName="payload-pre-processing" Oct 30 22:45:00.179821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179787 5226 cpu_manager.go:414] "containerMap: removing container" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" Oct 30 22:45:00.179821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179796 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" Oct 30 22:45:00.180025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179918 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b313574-95b0-4a9d-80f3-37c68537b527" containerName="istio-proxy" Oct 30 22:45:00.180025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179930 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a86fe5-a6d3-4b72-869c-e983f3d693d9" containerName="payload-pre-processing" Oct 30 22:45:00.180025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.179938 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afd7e57-39d4-4adb-8306-8587cdc2a13e" containerName="main" Oct 30 22:45:00.183546 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.183523 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:00.200877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.200851 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96"] Oct 30 22:45:00.207286 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.207268 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs"] Oct 30 22:45:00.207411 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.207394 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:00.210905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.210879 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:00.224676 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.224657 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm"] Oct 30 22:45:00.228106 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.228091 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:00.244915 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.244898 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\"" Oct 30 22:45:00.257467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.257447 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e198900-f639-4ff1-98d3-4f3442f3b785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx\" (UID: \"8e198900-f639-4ff1-98d3-4f3442f3b785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:00.257589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.257572 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm7lm\" (UniqueName: \"kubernetes.io/projected/8e198900-f639-4ff1-98d3-4f3442f3b785-kube-api-access-jm7lm\") pod \"maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx\" (UID: \"8e198900-f639-4ff1-98d3-4f3442f3b785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:00.284659 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.284630 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx"] Oct 30 22:45:00.331794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.331764 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96"] Oct 30 22:45:00.333497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.333471 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs"] Oct 30 22:45:00.358966 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.358931 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04469b0c-07cf-497c-9cbc-5233a2e7a08c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-6nzbs\" (UID: \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:00.359108 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.358980 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfd7\" (UniqueName: \"kubernetes.io/projected/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-kube-api-access-2wfd7\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm\" (UID: \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:00.359108 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.359059 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e198900-f639-4ff1-98d3-4f3442f3b785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx\" (UID: \"8e198900-f639-4ff1-98d3-4f3442f3b785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:00.359253 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.359130 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fpr\" (UniqueName: \"kubernetes.io/projected/ce6984d6-ca74-4760-9038-6a0d1faa77df-kube-api-access-z2fpr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96\" (UID: \"ce6984d6-ca74-4760-9038-6a0d1faa77df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:00.359253 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.359177 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm7lm\" (UniqueName: \"kubernetes.io/projected/8e198900-f639-4ff1-98d3-4f3442f3b785-kube-api-access-jm7lm\") pod \"maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx\" (UID: \"8e198900-f639-4ff1-98d3-4f3442f3b785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:00.359377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.359297 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcx5p\" (UniqueName: \"kubernetes.io/projected/04469b0c-07cf-497c-9cbc-5233a2e7a08c-kube-api-access-fcx5p\") pod \"maas-api-key-cleanup-29890005-6nzbs\" (UID: \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:00.359377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.359354 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6984d6-ca74-4760-9038-6a0d1faa77df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96\" (UID: \"ce6984d6-ca74-4760-9038-6a0d1faa77df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:00.359486 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.359442 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm\" (UID: \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:00.359817 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.359796 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e198900-f639-4ff1-98d3-4f3442f3b785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx\" (UID: \"8e198900-f639-4ff1-98d3-4f3442f3b785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:00.399876 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.399849 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm"] Oct 30 22:45:00.418086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.418061 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm7lm\" (UniqueName: \"kubernetes.io/projected/8e198900-f639-4ff1-98d3-4f3442f3b785-kube-api-access-jm7lm\") pod \"maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx\" (UID: \"8e198900-f639-4ff1-98d3-4f3442f3b785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:00.460901 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.460834 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcx5p\" (UniqueName: \"kubernetes.io/projected/04469b0c-07cf-497c-9cbc-5233a2e7a08c-kube-api-access-fcx5p\") pod \"maas-api-key-cleanup-29890005-6nzbs\" (UID: \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:00.460901 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.460881 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6984d6-ca74-4760-9038-6a0d1faa77df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96\" (UID: \"ce6984d6-ca74-4760-9038-6a0d1faa77df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:00.461147 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.460908 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm\" (UID: \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:00.461147 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.460959 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04469b0c-07cf-497c-9cbc-5233a2e7a08c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-6nzbs\" (UID: \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:00.461147 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.460977 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfd7\" (UniqueName: \"kubernetes.io/projected/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-kube-api-access-2wfd7\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm\" (UID: \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:00.461147 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.461048 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fpr\" (UniqueName: \"kubernetes.io/projected/ce6984d6-ca74-4760-9038-6a0d1faa77df-kube-api-access-z2fpr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96\" (UID: \"ce6984d6-ca74-4760-9038-6a0d1faa77df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:00.461582 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.461559 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6984d6-ca74-4760-9038-6a0d1faa77df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96\" (UID: \"ce6984d6-ca74-4760-9038-6a0d1faa77df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:00.461698 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.461677 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm\" (UID: \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:00.461825 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.461808 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04469b0c-07cf-497c-9cbc-5233a2e7a08c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-6nzbs\" (UID: \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:00.497985 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.497962 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:00.519407 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.519379 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fpr\" (UniqueName: \"kubernetes.io/projected/ce6984d6-ca74-4760-9038-6a0d1faa77df-kube-api-access-z2fpr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96\" (UID: \"ce6984d6-ca74-4760-9038-6a0d1faa77df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:00.521710 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.521686 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:00.522136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.522096 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfd7\" (UniqueName: \"kubernetes.io/projected/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-kube-api-access-2wfd7\") pod \"maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm\" (UID: \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:00.522866 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.522845 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcx5p\" (UniqueName: \"kubernetes.io/projected/04469b0c-07cf-497c-9cbc-5233a2e7a08c-kube-api-access-fcx5p\") pod \"maas-api-key-cleanup-29890005-6nzbs\" (UID: \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:00.528916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.527568 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:00.542251 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.540367 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:00.883994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.883915 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx"] Oct 30 22:45:00.890389 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:45:00.890358 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e198900_f639_4ff1_98d3_4f3442f3b785.slice/crio-25528ad7e55306ddcc668378bf0af2625df05e79ea7dcdb0d363c226ed1bf5c0 WatchSource:0}: Error finding container 25528ad7e55306ddcc668378bf0af2625df05e79ea7dcdb0d363c226ed1bf5c0: Status 404 returned error can't find the container with id 25528ad7e55306ddcc668378bf0af2625df05e79ea7dcdb0d363c226ed1bf5c0 Oct 30 22:45:00.934608 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.934578 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96"] Oct 30 22:45:00.940260 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:45:00.940230 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6984d6_ca74_4760_9038_6a0d1faa77df.slice/crio-1c18578d0c62e71fce1f70f61379e2ddb9aef0570bb9c24a09a5b4b2610b7e10 WatchSource:0}: Error finding container 1c18578d0c62e71fce1f70f61379e2ddb9aef0570bb9c24a09a5b4b2610b7e10: Status 404 returned error can't find the container with id 1c18578d0c62e71fce1f70f61379e2ddb9aef0570bb9c24a09a5b4b2610b7e10 Oct 30 22:45:00.965493 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.965468 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm"] Oct 30 22:45:00.972340 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:45:00.972318 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57cf677_fcd9_4efc_8a1b_9c3c9770a1e2.slice/crio-39aef966715ddccbb9ac4f4356adb73ad62bf0104cd3875bf4fd2cec52f4433e WatchSource:0}: Error finding container 39aef966715ddccbb9ac4f4356adb73ad62bf0104cd3875bf4fd2cec52f4433e: Status 404 returned error can't find the container with id 39aef966715ddccbb9ac4f4356adb73ad62bf0104cd3875bf4fd2cec52f4433e Oct 30 22:45:00.985535 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:00.985510 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs"] Oct 30 22:45:00.990434 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:45:00.990409 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04469b0c_07cf_497c_9cbc_5233a2e7a08c.slice/crio-ab8b5602781a2acb20dd4552aa860f5324614aff861c54bf53d61e77ed363a49 WatchSource:0}: Error finding container ab8b5602781a2acb20dd4552aa860f5324614aff861c54bf53d61e77ed363a49: Status 404 returned error can't find the container with id ab8b5602781a2acb20dd4552aa860f5324614aff861c54bf53d61e77ed363a49 Oct 30 22:45:01.076994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:01.076959 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" event={"ID":"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2","Type":"ContainerStarted","Data":"39aef966715ddccbb9ac4f4356adb73ad62bf0104cd3875bf4fd2cec52f4433e"} Oct 30 22:45:01.078551 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:01.078523 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" event={"ID":"8e198900-f639-4ff1-98d3-4f3442f3b785","Type":"ContainerStarted","Data":"25528ad7e55306ddcc668378bf0af2625df05e79ea7dcdb0d363c226ed1bf5c0"} Oct 30 22:45:01.079785 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:01.079762 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" event={"ID":"04469b0c-07cf-497c-9cbc-5233a2e7a08c","Type":"ContainerStarted","Data":"ab8b5602781a2acb20dd4552aa860f5324614aff861c54bf53d61e77ed363a49"} Oct 30 22:45:01.080855 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:01.080832 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" event={"ID":"ce6984d6-ca74-4760-9038-6a0d1faa77df","Type":"ContainerStarted","Data":"1c18578d0c62e71fce1f70f61379e2ddb9aef0570bb9c24a09a5b4b2610b7e10"} Oct 30 22:45:02.088691 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.088627 5226 generic.go:356] "Generic (PLEG): container finished" podID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerID="31c1547e29a9972ca94ed2bb1b256912df676365749c9f835d46cc2c95add2bd" exitCode=60 Oct 30 22:45:02.089047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.088712 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" event={"ID":"8e198900-f639-4ff1-98d3-4f3442f3b785","Type":"ContainerDied","Data":"31c1547e29a9972ca94ed2bb1b256912df676365749c9f835d46cc2c95add2bd"} Oct 30 22:45:02.089047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.088939 5226 scope.go:122] "RemoveContainer" containerID="31c1547e29a9972ca94ed2bb1b256912df676365749c9f835d46cc2c95add2bd" Oct 30 22:45:02.090413 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.090385 5226 generic.go:356] "Generic (PLEG): container finished" podID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerID="da555340d70ccff10030b54b3cf6028bd9f3fa862503cc6bf1d92dc4374eca21" exitCode=60 Oct 30 22:45:02.090525 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.090418 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" event={"ID":"04469b0c-07cf-497c-9cbc-5233a2e7a08c","Type":"ContainerDied","Data":"da555340d70ccff10030b54b3cf6028bd9f3fa862503cc6bf1d92dc4374eca21"} Oct 30 22:45:02.090743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.090727 5226 scope.go:122] "RemoveContainer" containerID="da555340d70ccff10030b54b3cf6028bd9f3fa862503cc6bf1d92dc4374eca21" Oct 30 22:45:02.092874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.092521 5226 generic.go:356] "Generic (PLEG): container finished" podID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerID="4540c17302af1ffbb4dba9a8307ebf60b0b111227f6bf3f31c4aeb4a614b7165" exitCode=60 Oct 30 22:45:02.092874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.092611 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" event={"ID":"ce6984d6-ca74-4760-9038-6a0d1faa77df","Type":"ContainerDied","Data":"4540c17302af1ffbb4dba9a8307ebf60b0b111227f6bf3f31c4aeb4a614b7165"} Oct 30 22:45:02.092959 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.092889 5226 scope.go:122] "RemoveContainer" containerID="4540c17302af1ffbb4dba9a8307ebf60b0b111227f6bf3f31c4aeb4a614b7165" Oct 30 22:45:02.095321 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.095282 5226 generic.go:356] "Generic (PLEG): container finished" podID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerID="53c9ac1bcd13d04999bd02d0e6769db002fc33e78eed9892a9dd0af702089991" exitCode=60 Oct 30 22:45:02.095400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.095340 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" event={"ID":"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2","Type":"ContainerDied","Data":"53c9ac1bcd13d04999bd02d0e6769db002fc33e78eed9892a9dd0af702089991"} Oct 30 22:45:02.095594 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:02.095575 5226 scope.go:122] "RemoveContainer" containerID="53c9ac1bcd13d04999bd02d0e6769db002fc33e78eed9892a9dd0af702089991" Oct 30 22:45:03.103747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.103715 5226 generic.go:356] "Generic (PLEG): container finished" podID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerID="56255e6fe568db1f29e9ea215758fa7c30cf597b60154b020c8c0536d8c5f8e3" exitCode=60 Oct 30 22:45:03.104174 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.103801 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" event={"ID":"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2","Type":"ContainerDied","Data":"56255e6fe568db1f29e9ea215758fa7c30cf597b60154b020c8c0536d8c5f8e3"} Oct 30 22:45:03.104174 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.103869 5226 scope.go:122] "RemoveContainer" containerID="53c9ac1bcd13d04999bd02d0e6769db002fc33e78eed9892a9dd0af702089991" Oct 30 22:45:03.104174 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.104088 5226 scope.go:122] "RemoveContainer" containerID="56255e6fe568db1f29e9ea215758fa7c30cf597b60154b020c8c0536d8c5f8e3" Oct 30 22:45:03.104444 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:03.104416 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm_odh-ai-gateway-infra(f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" Oct 30 22:45:03.105897 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.105875 5226 generic.go:356] "Generic (PLEG): container finished" podID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerID="97975804bc5c6e4e6f9d2859b42fdc2814903fae3e9b19c0bd63233f0d3b3dba" exitCode=60 Oct 30 22:45:03.105987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.105920 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" event={"ID":"8e198900-f639-4ff1-98d3-4f3442f3b785","Type":"ContainerDied","Data":"97975804bc5c6e4e6f9d2859b42fdc2814903fae3e9b19c0bd63233f0d3b3dba"} Oct 30 22:45:03.106164 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.106149 5226 scope.go:122] "RemoveContainer" containerID="97975804bc5c6e4e6f9d2859b42fdc2814903fae3e9b19c0bd63233f0d3b3dba" Oct 30 22:45:03.106420 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:03.106398 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx_odh-ai-gateway-infra(8e198900-f639-4ff1-98d3-4f3442f3b785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" Oct 30 22:45:03.107695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.107673 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" event={"ID":"04469b0c-07cf-497c-9cbc-5233a2e7a08c","Type":"ContainerStarted","Data":"180eafea6718157fab18c9ad41c6a82295c6c85d6ab086d294e4495caff5bfdc"} Oct 30 22:45:03.109761 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.109738 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" event={"ID":"ce6984d6-ca74-4760-9038-6a0d1faa77df","Type":"ContainerStarted","Data":"cc8d1ef7138ee2a6bdb1d389f49085b8fcdecd4a670aeca161056701eb18ed09"} Oct 30 22:45:03.118261 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.118240 5226 scope.go:122] "RemoveContainer" containerID="31c1547e29a9972ca94ed2bb1b256912df676365749c9f835d46cc2c95add2bd" Oct 30 22:45:03.324136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.324086 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" podStartSLOduration=3.324074263 podStartE2EDuration="3.324074263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.323564237 +0000 UTC m=+2594.580905946" watchObservedRunningTime="2026-10-30 22:45:03.324074263 +0000 UTC m=+2594.581415964" Oct 30 22:45:03.544356 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:03.544249 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" podStartSLOduration=3.544235398 podStartE2EDuration="3.544235398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.541463944 +0000 UTC m=+2594.798805651" watchObservedRunningTime="2026-10-30 22:45:03.544235398 +0000 UTC m=+2594.801577108" Oct 30 22:45:04.117153 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.117118 5226 generic.go:356] "Generic (PLEG): container finished" podID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerID="180eafea6718157fab18c9ad41c6a82295c6c85d6ab086d294e4495caff5bfdc" exitCode=60 Oct 30 22:45:04.117669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.117194 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" event={"ID":"04469b0c-07cf-497c-9cbc-5233a2e7a08c","Type":"ContainerDied","Data":"180eafea6718157fab18c9ad41c6a82295c6c85d6ab086d294e4495caff5bfdc"} Oct 30 22:45:04.117669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.117281 5226 scope.go:122] "RemoveContainer" containerID="da555340d70ccff10030b54b3cf6028bd9f3fa862503cc6bf1d92dc4374eca21" Oct 30 22:45:04.117669 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.117518 5226 scope.go:122] "RemoveContainer" containerID="180eafea6718157fab18c9ad41c6a82295c6c85d6ab086d294e4495caff5bfdc" Oct 30 22:45:04.117822 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:04.117794 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-6nzbs_odh-ai-gateway-infra(04469b0c-07cf-497c-9cbc-5233a2e7a08c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" Oct 30 22:45:04.119293 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.119268 5226 generic.go:356] "Generic (PLEG): container finished" podID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerID="cc8d1ef7138ee2a6bdb1d389f49085b8fcdecd4a670aeca161056701eb18ed09" exitCode=60 Oct 30 22:45:04.119397 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.119310 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" event={"ID":"ce6984d6-ca74-4760-9038-6a0d1faa77df","Type":"ContainerDied","Data":"cc8d1ef7138ee2a6bdb1d389f49085b8fcdecd4a670aeca161056701eb18ed09"} Oct 30 22:45:04.119578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.119564 5226 scope.go:122] "RemoveContainer" containerID="cc8d1ef7138ee2a6bdb1d389f49085b8fcdecd4a670aeca161056701eb18ed09" Oct 30 22:45:04.119855 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:04.119838 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96_odh-ai-gateway-infra(ce6984d6-ca74-4760-9038-6a0d1faa77df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" Oct 30 22:45:04.121187 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.121169 5226 scope.go:122] "RemoveContainer" containerID="56255e6fe568db1f29e9ea215758fa7c30cf597b60154b020c8c0536d8c5f8e3" Oct 30 22:45:04.121470 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:04.121437 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm_odh-ai-gateway-infra(f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" Oct 30 22:45:04.122713 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.122696 5226 scope.go:122] "RemoveContainer" containerID="97975804bc5c6e4e6f9d2859b42fdc2814903fae3e9b19c0bd63233f0d3b3dba" Oct 30 22:45:04.122940 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:04.122921 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx_odh-ai-gateway-infra(8e198900-f639-4ff1-98d3-4f3442f3b785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" Oct 30 22:45:04.129466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:04.129436 5226 scope.go:122] "RemoveContainer" containerID="4540c17302af1ffbb4dba9a8307ebf60b0b111227f6bf3f31c4aeb4a614b7165" Oct 30 22:45:05.129970 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:05.129938 5226 scope.go:122] "RemoveContainer" containerID="cc8d1ef7138ee2a6bdb1d389f49085b8fcdecd4a670aeca161056701eb18ed09" Oct 30 22:45:05.130474 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:05.130307 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96_odh-ai-gateway-infra(ce6984d6-ca74-4760-9038-6a0d1faa77df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" Oct 30 22:45:05.131641 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:05.131613 5226 scope.go:122] "RemoveContainer" containerID="180eafea6718157fab18c9ad41c6a82295c6c85d6ab086d294e4495caff5bfdc" Oct 30 22:45:05.131850 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:05.131832 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-6nzbs_odh-ai-gateway-infra(04469b0c-07cf-497c-9cbc-5233a2e7a08c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" Oct 30 22:45:13.898659 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:13.898625 5226 scope.go:122] "RemoveContainer" containerID="97975804bc5c6e4e6f9d2859b42fdc2814903fae3e9b19c0bd63233f0d3b3dba" Oct 30 22:45:15.192874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:15.192841 5226 generic.go:356] "Generic (PLEG): container finished" podID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerID="654e98898008849753f87b7a57295cb41fd06aa75ab164954af484073bbb8153" exitCode=60 Oct 30 22:45:15.193414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:15.192893 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" event={"ID":"8e198900-f639-4ff1-98d3-4f3442f3b785","Type":"ContainerDied","Data":"654e98898008849753f87b7a57295cb41fd06aa75ab164954af484073bbb8153"} Oct 30 22:45:15.193414 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:15.192944 5226 scope.go:122] "RemoveContainer" containerID="97975804bc5c6e4e6f9d2859b42fdc2814903fae3e9b19c0bd63233f0d3b3dba" Oct 30 22:45:15.193529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:15.193421 5226 scope.go:122] "RemoveContainer" containerID="654e98898008849753f87b7a57295cb41fd06aa75ab164954af484073bbb8153" Oct 30 22:45:15.193781 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:15.193753 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx_odh-ai-gateway-infra(8e198900-f639-4ff1-98d3-4f3442f3b785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" Oct 30 22:45:15.898280 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:15.898253 5226 scope.go:122] "RemoveContainer" containerID="cc8d1ef7138ee2a6bdb1d389f49085b8fcdecd4a670aeca161056701eb18ed09" Oct 30 22:45:16.311887 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:16.311853 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx"] Oct 30 22:45:16.464875 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:16.464849 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:16.543253 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:16.543142 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e198900-f639-4ff1-98d3-4f3442f3b785-kube-api-access-jm7lm\" (UniqueName: \"kubernetes.io/projected/8e198900-f639-4ff1-98d3-4f3442f3b785-kube-api-access-jm7lm\") pod \"8e198900-f639-4ff1-98d3-4f3442f3b785\" (UID: \"8e198900-f639-4ff1-98d3-4f3442f3b785\") " Oct 30 22:45:16.543253 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:16.543181 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e198900-f639-4ff1-98d3-4f3442f3b785-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e198900-f639-4ff1-98d3-4f3442f3b785-maas-api-service-ca\") pod \"8e198900-f639-4ff1-98d3-4f3442f3b785\" (UID: \"8e198900-f639-4ff1-98d3-4f3442f3b785\") " Oct 30 22:45:16.543650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:16.543621 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e198900-f639-4ff1-98d3-4f3442f3b785-maas-api-service-ca" pod "8e198900-f639-4ff1-98d3-4f3442f3b785" (UID: "8e198900-f639-4ff1-98d3-4f3442f3b785"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:16.543768 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:16.543735 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e198900-f639-4ff1-98d3-4f3442f3b785-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:16.545232 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:16.545199 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e198900-f639-4ff1-98d3-4f3442f3b785-kube-api-access-jm7lm" pod "8e198900-f639-4ff1-98d3-4f3442f3b785" (UID: "8e198900-f639-4ff1-98d3-4f3442f3b785"). InnerVolumeSpecName "kube-api-access-jm7lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:16.645380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:16.645349 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm7lm\" (UniqueName: \"kubernetes.io/projected/8e198900-f639-4ff1-98d3-4f3442f3b785-kube-api-access-jm7lm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.207506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.207473 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" Oct 30 22:45:17.207716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.207505 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx" event={"ID":"8e198900-f639-4ff1-98d3-4f3442f3b785","Type":"ContainerDied","Data":"25528ad7e55306ddcc668378bf0af2625df05e79ea7dcdb0d363c226ed1bf5c0"} Oct 30 22:45:17.207716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.207593 5226 scope.go:122] "RemoveContainer" containerID="654e98898008849753f87b7a57295cb41fd06aa75ab164954af484073bbb8153" Oct 30 22:45:17.209568 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.209537 5226 generic.go:356] "Generic (PLEG): container finished" podID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerID="c785941946f4a37556459ee49ca3b6f80ea580df4744b7e3b0e0c4df7539dea6" exitCode=60 Oct 30 22:45:17.209700 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.209618 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" event={"ID":"ce6984d6-ca74-4760-9038-6a0d1faa77df","Type":"ContainerDied","Data":"c785941946f4a37556459ee49ca3b6f80ea580df4744b7e3b0e0c4df7539dea6"} Oct 30 22:45:17.210083 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.210063 5226 scope.go:122] "RemoveContainer" containerID="c785941946f4a37556459ee49ca3b6f80ea580df4744b7e3b0e0c4df7539dea6" Oct 30 22:45:17.210551 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:17.210410 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96_odh-ai-gateway-infra(ce6984d6-ca74-4760-9038-6a0d1faa77df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" Oct 30 22:45:17.217718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.217699 5226 scope.go:122] "RemoveContainer" containerID="cc8d1ef7138ee2a6bdb1d389f49085b8fcdecd4a670aeca161056701eb18ed09" Oct 30 22:45:17.472184 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.472074 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx"] Oct 30 22:45:17.474644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.474617 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-afb98c-29890005-bw6mx"] Oct 30 22:45:17.898101 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.898022 5226 scope.go:122] "RemoveContainer" containerID="180eafea6718157fab18c9ad41c6a82295c6c85d6ab086d294e4495caff5bfdc" Oct 30 22:45:17.898296 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:17.898267 5226 scope.go:122] "RemoveContainer" containerID="56255e6fe568db1f29e9ea215758fa7c30cf597b60154b020c8c0536d8c5f8e3" Oct 30 22:45:18.382930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.382898 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96"] Oct 30 22:45:18.523840 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.523813 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:18.570047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.570017 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce6984d6-ca74-4760-9038-6a0d1faa77df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6984d6-ca74-4760-9038-6a0d1faa77df-maas-api-service-ca\") pod \"ce6984d6-ca74-4760-9038-6a0d1faa77df\" (UID: \"ce6984d6-ca74-4760-9038-6a0d1faa77df\") " Oct 30 22:45:18.570283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.570134 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6984d6-ca74-4760-9038-6a0d1faa77df-kube-api-access-z2fpr\" (UniqueName: \"kubernetes.io/projected/ce6984d6-ca74-4760-9038-6a0d1faa77df-kube-api-access-z2fpr\") pod \"ce6984d6-ca74-4760-9038-6a0d1faa77df\" (UID: \"ce6984d6-ca74-4760-9038-6a0d1faa77df\") " Oct 30 22:45:18.570447 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.570419 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6984d6-ca74-4760-9038-6a0d1faa77df-maas-api-service-ca" pod "ce6984d6-ca74-4760-9038-6a0d1faa77df" (UID: "ce6984d6-ca74-4760-9038-6a0d1faa77df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:18.570590 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.570574 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6984d6-ca74-4760-9038-6a0d1faa77df-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.572252 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.572232 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6984d6-ca74-4760-9038-6a0d1faa77df-kube-api-access-z2fpr" pod "ce6984d6-ca74-4760-9038-6a0d1faa77df" (UID: "ce6984d6-ca74-4760-9038-6a0d1faa77df"). InnerVolumeSpecName "kube-api-access-z2fpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:18.672328 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.672290 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2fpr\" (UniqueName: \"kubernetes.io/projected/ce6984d6-ca74-4760-9038-6a0d1faa77df-kube-api-access-z2fpr\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.907064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:18.907031 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" path="/var/lib/kubelet/pods/8e198900-f639-4ff1-98d3-4f3442f3b785/volumes" Oct 30 22:45:19.225578 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.225543 5226 generic.go:356] "Generic (PLEG): container finished" podID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerID="9b56f2da84613d5c350070894fe81844f0a369a7aca9533c05067bc32302ea4c" exitCode=60 Oct 30 22:45:19.225774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.225619 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" event={"ID":"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2","Type":"ContainerDied","Data":"9b56f2da84613d5c350070894fe81844f0a369a7aca9533c05067bc32302ea4c"} Oct 30 22:45:19.225774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.225726 5226 scope.go:122] "RemoveContainer" containerID="56255e6fe568db1f29e9ea215758fa7c30cf597b60154b020c8c0536d8c5f8e3" Oct 30 22:45:19.226150 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.226126 5226 scope.go:122] "RemoveContainer" containerID="9b56f2da84613d5c350070894fe81844f0a369a7aca9533c05067bc32302ea4c" Oct 30 22:45:19.226498 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:19.226453 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm_odh-ai-gateway-infra(f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" Oct 30 22:45:19.228152 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.228131 5226 generic.go:356] "Generic (PLEG): container finished" podID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerID="676476bb273c5e96975db5567130140be0d640d490e54a5cc0dc51d8766b8ed7" exitCode=60 Oct 30 22:45:19.228273 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.228230 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" event={"ID":"04469b0c-07cf-497c-9cbc-5233a2e7a08c","Type":"ContainerDied","Data":"676476bb273c5e96975db5567130140be0d640d490e54a5cc0dc51d8766b8ed7"} Oct 30 22:45:19.228662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.228645 5226 scope.go:122] "RemoveContainer" containerID="676476bb273c5e96975db5567130140be0d640d490e54a5cc0dc51d8766b8ed7" Oct 30 22:45:19.228978 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:19.228958 5226 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-6nzbs_odh-ai-gateway-infra(04469b0c-07cf-497c-9cbc-5233a2e7a08c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" Oct 30 22:45:19.229694 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.229671 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" event={"ID":"ce6984d6-ca74-4760-9038-6a0d1faa77df","Type":"ContainerDied","Data":"1c18578d0c62e71fce1f70f61379e2ddb9aef0570bb9c24a09a5b4b2610b7e10"} Oct 30 22:45:19.229790 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.229722 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96" Oct 30 22:45:19.242777 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.242758 5226 scope.go:122] "RemoveContainer" containerID="180eafea6718157fab18c9ad41c6a82295c6c85d6ab086d294e4495caff5bfdc" Oct 30 22:45:19.256696 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.256673 5226 scope.go:122] "RemoveContainer" containerID="c785941946f4a37556459ee49ca3b6f80ea580df4744b7e3b0e0c4df7539dea6" Oct 30 22:45:19.438601 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.438562 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96"] Oct 30 22:45:19.443146 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:19.443106 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5c2f68-29890005-tmm96"] Oct 30 22:45:20.343226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.343185 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm"] Oct 30 22:45:20.482071 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.482045 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:20.559671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.559633 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs"] Oct 30 22:45:20.598422 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.598347 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-kube-api-access-2wfd7\" (UniqueName: \"kubernetes.io/projected/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-kube-api-access-2wfd7\") pod \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\" (UID: \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\") " Oct 30 22:45:20.598594 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.598475 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-maas-api-service-ca\") pod \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\" (UID: \"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2\") " Oct 30 22:45:20.599266 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.599224 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-maas-api-service-ca" pod "f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" (UID: "f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:20.601596 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.601570 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-kube-api-access-2wfd7" pod "f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" (UID: "f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2"). InnerVolumeSpecName "kube-api-access-2wfd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:20.699903 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.699873 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.699903 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.699900 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfd7\" (UniqueName: \"kubernetes.io/projected/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2-kube-api-access-2wfd7\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.727456 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.727426 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:20.800338 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.800301 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04469b0c-07cf-497c-9cbc-5233a2e7a08c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04469b0c-07cf-497c-9cbc-5233a2e7a08c-maas-api-service-ca\") pod \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\" (UID: \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\") " Oct 30 22:45:20.800531 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.800507 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04469b0c-07cf-497c-9cbc-5233a2e7a08c-kube-api-access-fcx5p\" (UniqueName: \"kubernetes.io/projected/04469b0c-07cf-497c-9cbc-5233a2e7a08c-kube-api-access-fcx5p\") pod \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\" (UID: \"04469b0c-07cf-497c-9cbc-5233a2e7a08c\") " Oct 30 22:45:20.800748 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.800718 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04469b0c-07cf-497c-9cbc-5233a2e7a08c-maas-api-service-ca" pod "04469b0c-07cf-497c-9cbc-5233a2e7a08c" (UID: "04469b0c-07cf-497c-9cbc-5233a2e7a08c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:20.801011 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.800993 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04469b0c-07cf-497c-9cbc-5233a2e7a08c-maas-api-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.802788 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.802755 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04469b0c-07cf-497c-9cbc-5233a2e7a08c-kube-api-access-fcx5p" pod "04469b0c-07cf-497c-9cbc-5233a2e7a08c" (UID: "04469b0c-07cf-497c-9cbc-5233a2e7a08c"). InnerVolumeSpecName "kube-api-access-fcx5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:20.902137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.902062 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcx5p\" (UniqueName: \"kubernetes.io/projected/04469b0c-07cf-497c-9cbc-5233a2e7a08c-kube-api-access-fcx5p\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:20.905373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:20.905345 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" path="/var/lib/kubelet/pods/ce6984d6-ca74-4760-9038-6a0d1faa77df/volumes" Oct 30 22:45:21.246691 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.246661 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" Oct 30 22:45:21.246905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.246683 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm" event={"ID":"f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2","Type":"ContainerDied","Data":"39aef966715ddccbb9ac4f4356adb73ad62bf0104cd3875bf4fd2cec52f4433e"} Oct 30 22:45:21.246905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.246782 5226 scope.go:122] "RemoveContainer" containerID="9b56f2da84613d5c350070894fe81844f0a369a7aca9533c05067bc32302ea4c" Oct 30 22:45:21.248279 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.248254 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" event={"ID":"04469b0c-07cf-497c-9cbc-5233a2e7a08c","Type":"ContainerDied","Data":"ab8b5602781a2acb20dd4552aa860f5324614aff861c54bf53d61e77ed363a49"} Oct 30 22:45:21.248370 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.248314 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs" Oct 30 22:45:21.257999 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.257977 5226 scope.go:122] "RemoveContainer" containerID="676476bb273c5e96975db5567130140be0d640d490e54a5cc0dc51d8766b8ed7" Oct 30 22:45:21.400254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.400222 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm"] Oct 30 22:45:21.406659 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.406636 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23f3d4-29890005-kzbrm"] Oct 30 22:45:21.509079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.508997 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs"] Oct 30 22:45:21.513307 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:21.513282 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-6nzbs"] Oct 30 22:45:22.905731 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:22.905688 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" path="/var/lib/kubelet/pods/04469b0c-07cf-497c-9cbc-5233a2e7a08c/volumes" Oct 30 22:45:22.906363 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:22.906342 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" path="/var/lib/kubelet/pods/f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2/volumes" Oct 30 22:45:24.411720 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:24.411682 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:45:34.757148 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:34.757112 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q"] Oct 30 22:45:34.757638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:34.757346 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" podUID="7e772e6b-7473-4cd2-b906-efeca3dc1842" containerName="maas-api" containerID="cri-o://9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9" gracePeriod=30 Oct 30 22:45:35.055339 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.055316 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:45:35.165755 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.165716 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-metrics-certs\") pod \"7e772e6b-7473-4cd2-b906-efeca3dc1842\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " Oct 30 22:45:35.165930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.165819 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls\") pod \"7e772e6b-7473-4cd2-b906-efeca3dc1842\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " Oct 30 22:45:35.165930 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.165922 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e772e6b-7473-4cd2-b906-efeca3dc1842-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e772e6b-7473-4cd2-b906-efeca3dc1842-openshift-service-ca\") pod \"7e772e6b-7473-4cd2-b906-efeca3dc1842\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " Oct 30 22:45:35.166061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.165951 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e772e6b-7473-4cd2-b906-efeca3dc1842-kube-api-access-45jjz\" (UniqueName: \"kubernetes.io/projected/7e772e6b-7473-4cd2-b906-efeca3dc1842-kube-api-access-45jjz\") pod \"7e772e6b-7473-4cd2-b906-efeca3dc1842\" (UID: \"7e772e6b-7473-4cd2-b906-efeca3dc1842\") " Oct 30 22:45:35.167247 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.167088 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e772e6b-7473-4cd2-b906-efeca3dc1842-openshift-service-ca" pod "7e772e6b-7473-4cd2-b906-efeca3dc1842" (UID: "7e772e6b-7473-4cd2-b906-efeca3dc1842"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:35.171681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.171639 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e772e6b-7473-4cd2-b906-efeca3dc1842-kube-api-access-45jjz" pod "7e772e6b-7473-4cd2-b906-efeca3dc1842" (UID: "7e772e6b-7473-4cd2-b906-efeca3dc1842"). InnerVolumeSpecName "kube-api-access-45jjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:35.172686 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.172661 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-metrics-certs" pod "7e772e6b-7473-4cd2-b906-efeca3dc1842" (UID: "7e772e6b-7473-4cd2-b906-efeca3dc1842"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:35.172987 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.172935 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls" pod "7e772e6b-7473-4cd2-b906-efeca3dc1842" (UID: "7e772e6b-7473-4cd2-b906-efeca3dc1842"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:35.268565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.268524 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e772e6b-7473-4cd2-b906-efeca3dc1842-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:35.268565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.268559 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45jjz\" (UniqueName: \"kubernetes.io/projected/7e772e6b-7473-4cd2-b906-efeca3dc1842-kube-api-access-45jjz\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:35.268565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.268571 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:35.268800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.268582 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e772e6b-7473-4cd2-b906-efeca3dc1842-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:35.333603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.333495 5226 generic.go:356] "Generic (PLEG): container finished" podID="7e772e6b-7473-4cd2-b906-efeca3dc1842" containerID="9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9" exitCode=0 Oct 30 22:45:35.333603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.333548 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" Oct 30 22:45:35.333603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.333543 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" event={"ID":"7e772e6b-7473-4cd2-b906-efeca3dc1842","Type":"ContainerDied","Data":"9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9"} Oct 30 22:45:35.333874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.333623 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q" event={"ID":"7e772e6b-7473-4cd2-b906-efeca3dc1842","Type":"ContainerDied","Data":"062ea501cd41af9acbf6403746385efe5c9c3b48585347154b26bc61f73192d1"} Oct 30 22:45:35.333874 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.333655 5226 scope.go:122] "RemoveContainer" containerID="9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9" Oct 30 22:45:35.344574 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.344535 5226 scope.go:122] "RemoveContainer" containerID="9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9" Oct 30 22:45:35.344869 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:35.344843 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9\": container with ID starting with 9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9 not found: ID does not exist" containerID="9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9" Oct 30 22:45:35.344967 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.344878 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9"} err="failed to get container status \"9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9\": rpc error: code = NotFound desc = could not find container \"9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9\": container with ID starting with 9337f04560386d3204a45f7d14de6fca4f7dded0b6134ea97fbe3623433f14b9 not found: ID does not exist" Oct 30 22:45:35.364179 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.364147 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k"] Oct 30 22:45:35.605410 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.605329 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q"] Oct 30 22:45:35.611415 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:35.611388 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9fce61-7f68d65588-s8c6q"] Oct 30 22:45:36.904554 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:36.904516 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e772e6b-7473-4cd2-b906-efeca3dc1842" path="/var/lib/kubelet/pods/7e772e6b-7473-4cd2-b906-efeca3dc1842/volumes" Oct 30 22:45:44.181745 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:44.181710 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:45:44.532983 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:44.532890 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m"] Oct 30 22:45:44.533143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:44.533100 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" containerID="cri-o://fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e" gracePeriod=30 Oct 30 22:45:46.479946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:46.479906 5226 patch_prober.go:29] interesting pod/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:45:46.480390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:46.479976 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:45:49.464667 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.464629 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d"] Oct 30 22:45:49.509334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.509300 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv"] Oct 30 22:45:49.511586 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.511555 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w"] Oct 30 22:45:49.808400 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.808364 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:45:49.825458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825427 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-istio-token\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-istio-token\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.825637 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825465 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7054201d-3c4a-4839-b282-295d853eb43c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7054201d-3c4a-4839-b282-295d853eb43c-istiod-ca-cert\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.825637 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825488 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7054201d-3c4a-4839-b282-295d853eb43c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7054201d-3c4a-4839-b282-295d853eb43c-istio-podinfo\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.825637 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825509 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-certs\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.825815 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825641 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-data\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.825815 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825761 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-kube-api-access-j6gf8\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-kube-api-access-j6gf8\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.825927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825815 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-envoy\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.825927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825853 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-socket\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.825927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825861 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-certs" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.825927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825908 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-credential-socket\") pod \"7054201d-3c4a-4839-b282-295d853eb43c\" (UID: \"7054201d-3c4a-4839-b282-295d853eb43c\") " Oct 30 22:45:49.826117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825941 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-data" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.826117 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.825892 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7054201d-3c4a-4839-b282-295d853eb43c-istiod-ca-cert" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:49.826237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.826136 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-socket" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.826304 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.826266 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-credential-socket" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.826488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.826468 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7054201d-3c4a-4839-b282-295d853eb43c-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.826563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.826495 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.826563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.826511 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.826563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.826528 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.826563 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.826544 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.827850 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.827830 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-envoy" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:49.827927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.827904 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7054201d-3c4a-4839-b282-295d853eb43c-istio-podinfo" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:45:49.828137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.828113 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-istio-token" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:49.828137 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.828118 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-kube-api-access-j6gf8" pod "7054201d-3c4a-4839-b282-295d853eb43c" (UID: "7054201d-3c4a-4839-b282-295d853eb43c"). InnerVolumeSpecName "kube-api-access-j6gf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:49.927518 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.927484 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7054201d-3c4a-4839-b282-295d853eb43c-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.927518 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.927518 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.927744 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.927535 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7054201d-3c4a-4839-b282-295d853eb43c-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:49.927744 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:49.927549 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6gf8\" (UniqueName: \"kubernetes.io/projected/7054201d-3c4a-4839-b282-295d853eb43c-kube-api-access-j6gf8\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.394882 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.394818 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" podUID="7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" containerName="main" containerID="cri-o://a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385" gracePeriod=45 Oct 30 22:45:50.421794 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.421764 5226 generic.go:356] "Generic (PLEG): container finished" podID="7054201d-3c4a-4839-b282-295d853eb43c" containerID="fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e" exitCode=0 Oct 30 22:45:50.421950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.421817 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" Oct 30 22:45:50.421950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.421818 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" event={"ID":"7054201d-3c4a-4839-b282-295d853eb43c","Type":"ContainerDied","Data":"fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e"} Oct 30 22:45:50.421950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.421945 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m" event={"ID":"7054201d-3c4a-4839-b282-295d853eb43c","Type":"ContainerDied","Data":"44747f333aa443e161480f4bbe949b8bbedb41240fd7ba5c30248f5508e13abd"} Oct 30 22:45:50.422101 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.421973 5226 scope.go:122] "RemoveContainer" containerID="fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e" Oct 30 22:45:50.506585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.506562 5226 scope.go:122] "RemoveContainer" containerID="fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e" Oct 30 22:45:50.507035 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:50.506987 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e\": container with ID starting with fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e not found: ID does not exist" containerID="fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e" Oct 30 22:45:50.507105 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.507048 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e"} err="failed to get container status \"fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e\": rpc error: code = NotFound desc = could not find container \"fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e\": container with ID starting with fe40cfb34a7d00e149e65ebb6ff8f39e6b9c468bb8fe74985573adfcd404482e not found: ID does not exist" Oct 30 22:45:50.584640 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.584581 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m"] Oct 30 22:45:50.588730 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.588682 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23f3d4-openshift-default-54d69d7599-8mh9m"] Oct 30 22:45:50.661617 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.661540 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:45:50.735872 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.735838 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tls-certs\") pod \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " Oct 30 22:45:50.736079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.735942 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-model-cache\") pod \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " Oct 30 22:45:50.736079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.735980 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-home\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-home\") pod \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " Oct 30 22:45:50.736079 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.736052 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-kube-api-access-gmsjw\" (UniqueName: \"kubernetes.io/projected/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-kube-api-access-gmsjw\") pod \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " Oct 30 22:45:50.736245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.736096 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tmp-dir\") pod \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " Oct 30 22:45:50.736245 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.736163 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-dshm\") pod \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\" (UID: \"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a\") " Oct 30 22:45:50.736344 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.736266 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-model-cache" pod "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" (UID: "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:50.736527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.736495 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tmp-dir" pod "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" (UID: "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:50.736612 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.736584 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-home" pod "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" (UID: "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:50.736704 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.736682 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.736757 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.736708 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.738598 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.738567 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-kube-api-access-gmsjw" pod "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" (UID: "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a"). InnerVolumeSpecName "kube-api-access-gmsjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:50.738701 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.738651 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tls-certs" pod "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" (UID: "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:45:50.739003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.738840 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-dshm" pod "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" (UID: "7d6b4c2c-a1f7-41f0-9705-10b4d666a11a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:50.841024 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.839938 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.841024 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.839986 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.841024 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.839996 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.841024 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.840006 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmsjw\" (UniqueName: \"kubernetes.io/projected/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a-kube-api-access-gmsjw\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:50.904671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:50.904632 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7054201d-3c4a-4839-b282-295d853eb43c" path="/var/lib/kubelet/pods/7054201d-3c4a-4839-b282-295d853eb43c/volumes" Oct 30 22:45:51.428561 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.428517 5226 generic.go:356] "Generic (PLEG): container finished" podID="7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" containerID="a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385" exitCode=0 Oct 30 22:45:51.428747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.428581 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" Oct 30 22:45:51.428747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.428604 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" event={"ID":"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a","Type":"ContainerDied","Data":"a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385"} Oct 30 22:45:51.428747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.428650 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k" event={"ID":"7d6b4c2c-a1f7-41f0-9705-10b4d666a11a","Type":"ContainerDied","Data":"68741722ffe1e6d39204ec756042a7f69584602f3ade4ac010f78d69faaff6a3"} Oct 30 22:45:51.428747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.428672 5226 scope.go:122] "RemoveContainer" containerID="a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385" Oct 30 22:45:51.438226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.438199 5226 scope.go:122] "RemoveContainer" containerID="a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385" Oct 30 22:45:51.438477 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:51.438459 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385\": container with ID starting with a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385 not found: ID does not exist" containerID="a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385" Oct 30 22:45:51.438537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.438486 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385"} err="failed to get container status \"a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385\": rpc error: code = NotFound desc = could not find container \"a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385\": container with ID starting with a9f5377d9d005ceb22ed266c7381dcfd0f708f299b77725130e9af6ac2ad1385 not found: ID does not exist" Oct 30 22:45:51.583075 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.583040 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k"] Oct 30 22:45:51.589661 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:51.589633 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d6b55f/test-model-d6b55f-kserve-5c774bfb59-sbg2k"] Oct 30 22:45:52.904848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:52.904806 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" path="/var/lib/kubelet/pods/7d6b4c2c-a1f7-41f0-9705-10b4d666a11a/volumes" Oct 30 22:45:54.412086 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:45:54.412045 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:45:57.343197 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:57.343160 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf"] Oct 30 22:45:57.444658 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:57.444620 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j"] Oct 30 22:45:59.729944 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:59.729901 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9"] Oct 30 22:45:59.730460 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:45:59.730115 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" containerID="cri-o://7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74" gracePeriod=30 Oct 30 22:46:01.272945 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:01.272916 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z"] Oct 30 22:46:02.343747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:02.343701 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" podUID="a7e743f3-5463-4ae1-9c8a-75a892fbaceb" containerName="payload-processing" containerID="cri-o://59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137" gracePeriod=25 Oct 30 22:46:02.445878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:02.445828 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" podUID="3575f09c-1832-468b-b938-1fd125084dce" containerName="payload-pre-processing" containerID="cri-o://eae287d37ea1a123f1dc742c407176339be3debe822ef3a7339e8a0caefa9fac" gracePeriod=25 Oct 30 22:46:04.485861 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:04.485822 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" containerID="cri-o://0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3" gracePeriod=45 Oct 30 22:46:04.535907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:04.535859 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" containerID="cri-o://808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c" gracePeriod=45 Oct 30 22:46:04.537108 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:04.537079 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" containerID="cri-o://6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b" gracePeriod=45 Oct 30 22:46:04.860857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:04.860833 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:46:04.864405 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:04.864383 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:46:04.868837 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:04.868815 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:46:04.985979 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:04.985949 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:46:05.001032 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.000957 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-tmp-dir\") pod \"515e746e-b44c-463a-bc1a-5e57dfb5b920\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " Oct 30 22:46:05.001300 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001275 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515e746e-b44c-463a-bc1a-5e57dfb5b920-kube-api-access-bn4rh\" (UniqueName: \"kubernetes.io/projected/515e746e-b44c-463a-bc1a-5e57dfb5b920-kube-api-access-bn4rh\") pod \"515e746e-b44c-463a-bc1a-5e57dfb5b920\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " Oct 30 22:46:05.001476 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001459 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-model-cache\") pod \"515e746e-b44c-463a-bc1a-5e57dfb5b920\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " Oct 30 22:46:05.001613 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001594 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18303bc1-aa75-4a32-9173-a546eced5ba9-tls-certs\" (UniqueName: \"kubernetes.io/secret/18303bc1-aa75-4a32-9173-a546eced5ba9-tls-certs\") pod \"18303bc1-aa75-4a32-9173-a546eced5ba9\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " Oct 30 22:46:05.001752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001713 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/515e746e-b44c-463a-bc1a-5e57dfb5b920-tls-certs\" (UniqueName: \"kubernetes.io/secret/515e746e-b44c-463a-bc1a-5e57dfb5b920-tls-certs\") pod \"515e746e-b44c-463a-bc1a-5e57dfb5b920\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " Oct 30 22:46:05.001881 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001859 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18303bc1-aa75-4a32-9173-a546eced5ba9-kube-api-access-gwh6x\" (UniqueName: \"kubernetes.io/projected/18303bc1-aa75-4a32-9173-a546eced5ba9-kube-api-access-gwh6x\") pod \"18303bc1-aa75-4a32-9173-a546eced5ba9\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " Oct 30 22:46:05.001949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001893 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-kube-api-access-5fgns\" (UniqueName: \"kubernetes.io/projected/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-kube-api-access-5fgns\") pod \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " Oct 30 22:46:05.001949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001923 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tmp-dir\") pod \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " Oct 30 22:46:05.002107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001960 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-dshm\") pod \"18303bc1-aa75-4a32-9173-a546eced5ba9\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " Oct 30 22:46:05.002107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002001 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-model-cache\") pod \"18303bc1-aa75-4a32-9173-a546eced5ba9\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " Oct 30 22:46:05.002107 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002073 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-home\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-home\") pod \"515e746e-b44c-463a-bc1a-5e57dfb5b920\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " Oct 30 22:46:05.002349 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002112 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tls-certs\") pod \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " Oct 30 22:46:05.002349 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002142 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-dshm\") pod \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " Oct 30 22:46:05.002349 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002186 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-tmp-dir\") pod \"18303bc1-aa75-4a32-9173-a546eced5ba9\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " Oct 30 22:46:05.002349 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002260 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-dshm\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-dshm\") pod \"515e746e-b44c-463a-bc1a-5e57dfb5b920\" (UID: \"515e746e-b44c-463a-bc1a-5e57dfb5b920\") " Oct 30 22:46:05.002580 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002358 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-home\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-home\") pod \"18303bc1-aa75-4a32-9173-a546eced5ba9\" (UID: \"18303bc1-aa75-4a32-9173-a546eced5ba9\") " Oct 30 22:46:05.002580 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002399 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-model-cache\") pod \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " Oct 30 22:46:05.002580 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002516 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-home\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-home\") pod \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\" (UID: \"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0\") " Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001282 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-tmp-dir" pod "515e746e-b44c-463a-bc1a-5e57dfb5b920" (UID: "515e746e-b44c-463a-bc1a-5e57dfb5b920"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002952 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-home" pod "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" (UID: "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.003078 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.003099 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.001696 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-model-cache" pod "515e746e-b44c-463a-bc1a-5e57dfb5b920" (UID: "515e746e-b44c-463a-bc1a-5e57dfb5b920"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.002369 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-model-cache" pod "18303bc1-aa75-4a32-9173-a546eced5ba9" (UID: "18303bc1-aa75-4a32-9173-a546eced5ba9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.003718 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-home" pod "18303bc1-aa75-4a32-9173-a546eced5ba9" (UID: "18303bc1-aa75-4a32-9173-a546eced5ba9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.003933 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-model-cache" pod "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" (UID: "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.004228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.004122 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tmp-dir" pod "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" (UID: "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.005131 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.005104 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-home" pod "515e746e-b44c-463a-bc1a-5e57dfb5b920" (UID: "515e746e-b44c-463a-bc1a-5e57dfb5b920"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.006183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.006157 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-kube-api-access-5fgns" pod "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" (UID: "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0"). InnerVolumeSpecName "kube-api-access-5fgns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:05.007257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.007190 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-dshm" pod "515e746e-b44c-463a-bc1a-5e57dfb5b920" (UID: "515e746e-b44c-463a-bc1a-5e57dfb5b920"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.010027 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.010004 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-tmp-dir" pod "18303bc1-aa75-4a32-9173-a546eced5ba9" (UID: "18303bc1-aa75-4a32-9173-a546eced5ba9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.011357 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.011320 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515e746e-b44c-463a-bc1a-5e57dfb5b920-tls-certs" pod "515e746e-b44c-463a-bc1a-5e57dfb5b920" (UID: "515e746e-b44c-463a-bc1a-5e57dfb5b920"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:05.011357 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.011339 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-dshm" pod "18303bc1-aa75-4a32-9173-a546eced5ba9" (UID: "18303bc1-aa75-4a32-9173-a546eced5ba9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.011357 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.011349 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18303bc1-aa75-4a32-9173-a546eced5ba9-kube-api-access-gwh6x" pod "18303bc1-aa75-4a32-9173-a546eced5ba9" (UID: "18303bc1-aa75-4a32-9173-a546eced5ba9"). InnerVolumeSpecName "kube-api-access-gwh6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:05.011533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.011358 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-dshm" pod "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" (UID: "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.011702 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.011679 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515e746e-b44c-463a-bc1a-5e57dfb5b920-kube-api-access-bn4rh" pod "515e746e-b44c-463a-bc1a-5e57dfb5b920" (UID: "515e746e-b44c-463a-bc1a-5e57dfb5b920"). InnerVolumeSpecName "kube-api-access-bn4rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:05.011789 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.011689 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18303bc1-aa75-4a32-9173-a546eced5ba9-tls-certs" pod "18303bc1-aa75-4a32-9173-a546eced5ba9" (UID: "18303bc1-aa75-4a32-9173-a546eced5ba9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:05.013380 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.013353 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tls-certs" pod "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" (UID: "2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:05.103884 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.103844 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-envoy\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.103919 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-socket\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.103972 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-certs\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104011 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-kube-api-access-b4wzh\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-kube-api-access-b4wzh\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104043 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-data\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104076 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104075 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-podinfo\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104195 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-credential-socket\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104303 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16c9ead-eb60-477e-b415-5a378862ffb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16c9ead-eb60-477e-b415-5a378862ffb5-istiod-ca-cert\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104353 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-token\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-token\") pod \"c16c9ead-eb60-477e-b415-5a378862ffb5\" (UID: \"c16c9ead-eb60-477e-b415-5a378862ffb5\") " Oct 30 22:46:05.104393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104359 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-certs" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.104681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104662 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-credential-socket" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104883 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104905 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104918 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104933 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104950 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104965 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4rh\" (UniqueName: \"kubernetes.io/projected/515e746e-b44c-463a-bc1a-5e57dfb5b920-kube-api-access-bn4rh\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104981 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.104994 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105009 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18303bc1-aa75-4a32-9173-a546eced5ba9-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105026 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515e746e-b44c-463a-bc1a-5e57dfb5b920-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105040 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwh6x\" (UniqueName: \"kubernetes.io/projected/18303bc1-aa75-4a32-9173-a546eced5ba9-kube-api-access-gwh6x\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105057 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105055 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgns\" (UniqueName: \"kubernetes.io/projected/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-kube-api-access-5fgns\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105069 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105081 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105094 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18303bc1-aa75-4a32-9173-a546eced5ba9-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105106 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515e746e-b44c-463a-bc1a-5e57dfb5b920-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105121 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105134 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.105814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.105399 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-data" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.106369 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.106340 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16c9ead-eb60-477e-b415-5a378862ffb5-istiod-ca-cert" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:05.106622 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.106592 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-socket" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.108898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.108856 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-kube-api-access-b4wzh" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "kube-api-access-b4wzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:05.108898 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.108883 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-token" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:05.109151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.109123 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-podinfo" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:46:05.109608 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.109580 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-envoy" pod "c16c9ead-eb60-477e-b415-5a378862ffb5" (UID: "c16c9ead-eb60-477e-b415-5a378862ffb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:05.206257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.206219 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c16c9ead-eb60-477e-b415-5a378862ffb5-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.206257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.206259 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.206505 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.206275 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.206505 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.206289 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.206505 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.206303 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4wzh\" (UniqueName: \"kubernetes.io/projected/c16c9ead-eb60-477e-b415-5a378862ffb5-kube-api-access-b4wzh\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.206505 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.206317 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.206505 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.206328 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c16c9ead-eb60-477e-b415-5a378862ffb5-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:05.514692 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.514660 5226 generic.go:356] "Generic (PLEG): container finished" podID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerID="6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b" exitCode=0 Oct 30 22:46:05.515151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.514714 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" Oct 30 22:46:05.515151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.514740 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" event={"ID":"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0","Type":"ContainerDied","Data":"6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b"} Oct 30 22:46:05.515151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.514777 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w" event={"ID":"2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0","Type":"ContainerDied","Data":"002ce3164b005f140e5a29f86196a80ca10ad6fc5d73b4ddc41bf24dda081faf"} Oct 30 22:46:05.515151 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.514793 5226 scope.go:122] "RemoveContainer" containerID="6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b" Oct 30 22:46:05.516175 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.516152 5226 generic.go:356] "Generic (PLEG): container finished" podID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerID="7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74" exitCode=0 Oct 30 22:46:05.516306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.516228 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" event={"ID":"c16c9ead-eb60-477e-b415-5a378862ffb5","Type":"ContainerDied","Data":"7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74"} Oct 30 22:46:05.516306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.516274 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" event={"ID":"c16c9ead-eb60-477e-b415-5a378862ffb5","Type":"ContainerDied","Data":"4b4dad667c79e32c1ac64bdcc3a0d7ef3a62933816c9f967e24e571d1a479ea6"} Oct 30 22:46:05.516306 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.516233 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9" Oct 30 22:46:05.518237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.518186 5226 generic.go:356] "Generic (PLEG): container finished" podID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerID="808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c" exitCode=0 Oct 30 22:46:05.518346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.518249 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" Oct 30 22:46:05.518346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.518245 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" event={"ID":"18303bc1-aa75-4a32-9173-a546eced5ba9","Type":"ContainerDied","Data":"808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c"} Oct 30 22:46:05.518472 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.518358 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv" event={"ID":"18303bc1-aa75-4a32-9173-a546eced5ba9","Type":"ContainerDied","Data":"65abeef86c7ca908d0c46e596ae11266d86436a1f6c62d37ab5817c2f767ba2f"} Oct 30 22:46:05.520003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.519979 5226 generic.go:356] "Generic (PLEG): container finished" podID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerID="0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3" exitCode=0 Oct 30 22:46:05.520111 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.520017 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" event={"ID":"515e746e-b44c-463a-bc1a-5e57dfb5b920","Type":"ContainerDied","Data":"0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3"} Oct 30 22:46:05.520111 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.520022 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" Oct 30 22:46:05.520251 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.520045 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d" event={"ID":"515e746e-b44c-463a-bc1a-5e57dfb5b920","Type":"ContainerDied","Data":"fe4365ccc20889370a48c260202de2fcdfcb60667fe7f50ed103a71f1c39c056"} Oct 30 22:46:05.526920 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.526805 5226 scope.go:122] "RemoveContainer" containerID="6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b" Oct 30 22:46:05.527194 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:05.527169 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b\": container with ID starting with 6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b not found: ID does not exist" containerID="6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b" Oct 30 22:46:05.527294 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.527238 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b"} err="failed to get container status \"6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b\": rpc error: code = NotFound desc = could not find container \"6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b\": container with ID starting with 6407d9af58fa74b34e69d1257800d7ad79f93a8dfc2b90a2e6e6a47aabb56f3b not found: ID does not exist" Oct 30 22:46:05.527294 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.527265 5226 scope.go:122] "RemoveContainer" containerID="7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74" Oct 30 22:46:05.536561 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.536544 5226 scope.go:122] "RemoveContainer" containerID="7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74" Oct 30 22:46:05.536816 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:05.536798 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74\": container with ID starting with 7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74 not found: ID does not exist" containerID="7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74" Oct 30 22:46:05.536890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.536825 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74"} err="failed to get container status \"7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74\": rpc error: code = NotFound desc = could not find container \"7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74\": container with ID starting with 7b968fdb7a8aec48f48bc3a67f7ccf91556384a6b41efd768cffc97d0e366f74 not found: ID does not exist" Oct 30 22:46:05.536890 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.536847 5226 scope.go:122] "RemoveContainer" containerID="808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c" Oct 30 22:46:05.544811 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.544794 5226 scope.go:122] "RemoveContainer" containerID="808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c" Oct 30 22:46:05.545083 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:05.545064 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c\": container with ID starting with 808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c not found: ID does not exist" containerID="808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c" Oct 30 22:46:05.545130 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.545088 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c"} err="failed to get container status \"808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c\": rpc error: code = NotFound desc = could not find container \"808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c\": container with ID starting with 808b181121ace45ffe4f49b9863bdfc7b3a1ea27e30352dd5ff95e766f95921c not found: ID does not exist" Oct 30 22:46:05.545130 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.545102 5226 scope.go:122] "RemoveContainer" containerID="0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3" Oct 30 22:46:05.552674 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.552650 5226 scope.go:122] "RemoveContainer" containerID="0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3" Oct 30 22:46:05.552939 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:05.552921 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3\": container with ID starting with 0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3 not found: ID does not exist" containerID="0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3" Oct 30 22:46:05.552991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.552945 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3"} err="failed to get container status \"0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3\": rpc error: code = NotFound desc = could not find container \"0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3\": container with ID starting with 0d41aedce5554b5707c2d0aa8a427cabec063c81808958645295002d582a4be3 not found: ID does not exist" Oct 30 22:46:05.668603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.668568 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv"] Oct 30 22:46:05.673159 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.673135 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/premium-w3-23f3d4-kserve-7cf8df4f8c-p6bxv"] Oct 30 22:46:05.775752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.775663 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9"] Oct 30 22:46:05.778852 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.778823 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eb4aa1-openshift-default-74c4b68945-z8hw9"] Oct 30 22:46:05.881348 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.881312 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w"] Oct 30 22:46:05.883318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.883291 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/simulator-w3-23f3d4-kserve-666c764954-vwh6w"] Oct 30 22:46:05.985242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.985197 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d"] Oct 30 22:46:05.990877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:05.990846 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-23f3d4/distinct-2-w3-23f3d4-kserve-59758b5589-2tw2d"] Oct 30 22:46:06.866950 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:06.866908 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" podUID="3575f09c-1832-468b-b938-1fd125084dce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:46:06.904718 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:06.904681 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" path="/var/lib/kubelet/pods/18303bc1-aa75-4a32-9173-a546eced5ba9/volumes" Oct 30 22:46:06.905138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:06.905124 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" path="/var/lib/kubelet/pods/2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0/volumes" Oct 30 22:46:06.905539 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:06.905526 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" path="/var/lib/kubelet/pods/515e746e-b44c-463a-bc1a-5e57dfb5b920/volumes" Oct 30 22:46:06.906256 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:06.906241 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" path="/var/lib/kubelet/pods/c16c9ead-eb60-477e-b415-5a378862ffb5/volumes" Oct 30 22:46:07.872048 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:07.872011 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" podUID="a7e743f3-5463-4ae1-9c8a-75a892fbaceb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:46:10.950682 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:10.950648 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:46:13.290548 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:13.290516 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4"] Oct 30 22:46:16.293009 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.292941 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" podUID="94109748-986f-4293-ad0a-8795650676e9" containerName="main" containerID="cri-o://128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0" gracePeriod=45 Oct 30 22:46:16.556138 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.556082 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:46:16.585549 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.585518 5226 generic.go:356] "Generic (PLEG): container finished" podID="94109748-986f-4293-ad0a-8795650676e9" containerID="128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0" exitCode=0 Oct 30 22:46:16.585701 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.585566 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" Oct 30 22:46:16.585701 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.585601 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" event={"ID":"94109748-986f-4293-ad0a-8795650676e9","Type":"ContainerDied","Data":"128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0"} Oct 30 22:46:16.585701 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.585644 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z" event={"ID":"94109748-986f-4293-ad0a-8795650676e9","Type":"ContainerDied","Data":"fa90780726ccbd2f2ff4fcdab46697259296dcb28f38340c474e7a28ce4771ff"} Oct 30 22:46:16.585701 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.585661 5226 scope.go:122] "RemoveContainer" containerID="128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0" Oct 30 22:46:16.594556 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.594538 5226 scope.go:122] "RemoveContainer" containerID="128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0" Oct 30 22:46:16.594812 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:16.594794 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0\": container with ID starting with 128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0 not found: ID does not exist" containerID="128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0" Oct 30 22:46:16.594872 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.594820 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0"} err="failed to get container status \"128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0\": rpc error: code = NotFound desc = could not find container \"128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0\": container with ID starting with 128944299feb9599dfcc9a0d82558fbfbfc0a14a095c82aa1644fa58625a71f0 not found: ID does not exist" Oct 30 22:46:16.625277 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625241 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94109748-986f-4293-ad0a-8795650676e9-kube-api-access-gb6fv\" (UniqueName: \"kubernetes.io/projected/94109748-986f-4293-ad0a-8795650676e9-kube-api-access-gb6fv\") pod \"94109748-986f-4293-ad0a-8795650676e9\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " Oct 30 22:46:16.625390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625285 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94109748-986f-4293-ad0a-8795650676e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/94109748-986f-4293-ad0a-8795650676e9-tls-certs\") pod \"94109748-986f-4293-ad0a-8795650676e9\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " Oct 30 22:46:16.625390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625340 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-home\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-home\") pod \"94109748-986f-4293-ad0a-8795650676e9\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " Oct 30 22:46:16.625390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625361 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-model-cache\") pod \"94109748-986f-4293-ad0a-8795650676e9\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " Oct 30 22:46:16.625390 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625377 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-dshm\") pod \"94109748-986f-4293-ad0a-8795650676e9\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " Oct 30 22:46:16.625611 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625532 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-tmp-dir\") pod \"94109748-986f-4293-ad0a-8795650676e9\" (UID: \"94109748-986f-4293-ad0a-8795650676e9\") " Oct 30 22:46:16.625667 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625607 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-model-cache" pod "94109748-986f-4293-ad0a-8795650676e9" (UID: "94109748-986f-4293-ad0a-8795650676e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:16.625667 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625623 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-home" pod "94109748-986f-4293-ad0a-8795650676e9" (UID: "94109748-986f-4293-ad0a-8795650676e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:16.625828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625803 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-tmp-dir" pod "94109748-986f-4293-ad0a-8795650676e9" (UID: "94109748-986f-4293-ad0a-8795650676e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:16.625967 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625950 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:16.626027 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625974 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:16.626027 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.625989 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:16.627581 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.627560 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-dshm" pod "94109748-986f-4293-ad0a-8795650676e9" (UID: "94109748-986f-4293-ad0a-8795650676e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:16.627774 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.627759 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94109748-986f-4293-ad0a-8795650676e9-kube-api-access-gb6fv" pod "94109748-986f-4293-ad0a-8795650676e9" (UID: "94109748-986f-4293-ad0a-8795650676e9"). InnerVolumeSpecName "kube-api-access-gb6fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:16.627821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.627778 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94109748-986f-4293-ad0a-8795650676e9-tls-certs" pod "94109748-986f-4293-ad0a-8795650676e9" (UID: "94109748-986f-4293-ad0a-8795650676e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:16.727389 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.727363 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb6fv\" (UniqueName: \"kubernetes.io/projected/94109748-986f-4293-ad0a-8795650676e9-kube-api-access-gb6fv\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:16.727389 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.727388 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94109748-986f-4293-ad0a-8795650676e9-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:16.727557 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.727397 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94109748-986f-4293-ad0a-8795650676e9-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:16.867707 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.867630 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" podUID="3575f09c-1832-468b-b938-1fd125084dce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:46:16.999052 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:16.999014 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z"] Oct 30 22:46:17.001585 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:17.001557 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d585d2/test-model-d585d2-kserve-7477dfffc9-zh48z"] Oct 30 22:46:17.872095 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:17.872056 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" podUID="a7e743f3-5463-4ae1-9c8a-75a892fbaceb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:46:18.291270 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.291236 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" podUID="ee04e125-52ff-47e2-b35d-94abe86c6ed1" containerName="payload-processing" containerID="cri-o://cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c" gracePeriod=25 Oct 30 22:46:18.551755 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.551697 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:46:18.599418 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.599381 5226 generic.go:356] "Generic (PLEG): container finished" podID="ee04e125-52ff-47e2-b35d-94abe86c6ed1" containerID="cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c" exitCode=0 Oct 30 22:46:18.599603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.599430 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" Oct 30 22:46:18.599603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.599470 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" event={"ID":"ee04e125-52ff-47e2-b35d-94abe86c6ed1","Type":"ContainerDied","Data":"cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c"} Oct 30 22:46:18.599603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.599540 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4" event={"ID":"ee04e125-52ff-47e2-b35d-94abe86c6ed1","Type":"ContainerDied","Data":"b6db175344c454d86ec81f0d0be3c1133ad238a69ca9eda613edd901349f0db4"} Oct 30 22:46:18.599603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.599572 5226 scope.go:122] "RemoveContainer" containerID="cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c" Oct 30 22:46:18.611308 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.611284 5226 scope.go:122] "RemoveContainer" containerID="cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c" Oct 30 22:46:18.611617 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:18.611594 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c\": container with ID starting with cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c not found: ID does not exist" containerID="cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c" Oct 30 22:46:18.611680 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.611626 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c"} err="failed to get container status \"cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c\": rpc error: code = NotFound desc = could not find container \"cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c\": container with ID starting with cc35628baea3d5005bcac3d30628d56b8b36a4b526083059c82c5a8de2e7d27c not found: ID does not exist" Oct 30 22:46:18.647090 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.647059 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee04e125-52ff-47e2-b35d-94abe86c6ed1-kube-api-access-x9txc\" (UniqueName: \"kubernetes.io/projected/ee04e125-52ff-47e2-b35d-94abe86c6ed1-kube-api-access-x9txc\") pod \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\" (UID: \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\") " Oct 30 22:46:18.647272 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.647252 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee04e125-52ff-47e2-b35d-94abe86c6ed1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee04e125-52ff-47e2-b35d-94abe86c6ed1-plugins-config-volume\") pod \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\" (UID: \"ee04e125-52ff-47e2-b35d-94abe86c6ed1\") " Oct 30 22:46:18.647608 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.647585 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee04e125-52ff-47e2-b35d-94abe86c6ed1-plugins-config-volume" pod "ee04e125-52ff-47e2-b35d-94abe86c6ed1" (UID: "ee04e125-52ff-47e2-b35d-94abe86c6ed1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:18.649246 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.649224 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee04e125-52ff-47e2-b35d-94abe86c6ed1-kube-api-access-x9txc" pod "ee04e125-52ff-47e2-b35d-94abe86c6ed1" (UID: "ee04e125-52ff-47e2-b35d-94abe86c6ed1"). InnerVolumeSpecName "kube-api-access-x9txc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:18.749135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.749097 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee04e125-52ff-47e2-b35d-94abe86c6ed1-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:18.749135 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.749133 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9txc\" (UniqueName: \"kubernetes.io/projected/ee04e125-52ff-47e2-b35d-94abe86c6ed1-kube-api-access-x9txc\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:18.904235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:18.904139 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94109748-986f-4293-ad0a-8795650676e9" path="/var/lib/kubelet/pods/94109748-986f-4293-ad0a-8795650676e9/volumes" Oct 30 22:46:19.014492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:19.014463 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4"] Oct 30 22:46:19.020725 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:19.020623 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6b55f-9d98cc4f-dcmt4"] Oct 30 22:46:20.904648 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:20.904610 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee04e125-52ff-47e2-b35d-94abe86c6ed1" path="/var/lib/kubelet/pods/ee04e125-52ff-47e2-b35d-94abe86c6ed1/volumes" Oct 30 22:46:22.593067 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.593032 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:46:22.631659 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.631623 5226 generic.go:356] "Generic (PLEG): container finished" podID="a7e743f3-5463-4ae1-9c8a-75a892fbaceb" containerID="59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137" exitCode=0 Oct 30 22:46:22.631813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.631663 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" Oct 30 22:46:22.631813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.631698 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" event={"ID":"a7e743f3-5463-4ae1-9c8a-75a892fbaceb","Type":"ContainerDied","Data":"59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137"} Oct 30 22:46:22.631813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.631749 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf" event={"ID":"a7e743f3-5463-4ae1-9c8a-75a892fbaceb","Type":"ContainerDied","Data":"91054eda83fe3391586464d2c8c47a68eac756d20fb5cc7b2af9c70f57ea9b76"} Oct 30 22:46:22.631813 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.631774 5226 scope.go:122] "RemoveContainer" containerID="59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137" Oct 30 22:46:22.633949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.633814 5226 generic.go:356] "Generic (PLEG): container finished" podID="3575f09c-1832-468b-b938-1fd125084dce" containerID="eae287d37ea1a123f1dc742c407176339be3debe822ef3a7339e8a0caefa9fac" exitCode=0 Oct 30 22:46:22.633949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.633892 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" event={"ID":"3575f09c-1832-468b-b938-1fd125084dce","Type":"ContainerDied","Data":"eae287d37ea1a123f1dc742c407176339be3debe822ef3a7339e8a0caefa9fac"} Oct 30 22:46:22.642540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.642513 5226 scope.go:122] "RemoveContainer" containerID="59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137" Oct 30 22:46:22.642811 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:22.642793 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137\": container with ID starting with 59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137 not found: ID does not exist" containerID="59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137" Oct 30 22:46:22.642888 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.642818 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137"} err="failed to get container status \"59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137\": rpc error: code = NotFound desc = could not find container \"59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137\": container with ID starting with 59c4aa3b330a904d873fa6d03372c68f926f9db787cb25948bdd3b672f534137 not found: ID does not exist" Oct 30 22:46:22.690216 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.690173 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-kube-api-access-7fr8z\" (UniqueName: \"kubernetes.io/projected/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-kube-api-access-7fr8z\") pod \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\" (UID: \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\") " Oct 30 22:46:22.690383 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.690353 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-plugins-config-volume\") pod \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\" (UID: \"a7e743f3-5463-4ae1-9c8a-75a892fbaceb\") " Oct 30 22:46:22.690655 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.690635 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-plugins-config-volume" pod "a7e743f3-5463-4ae1-9c8a-75a892fbaceb" (UID: "a7e743f3-5463-4ae1-9c8a-75a892fbaceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:22.690800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.690779 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:22.692264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.692238 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-kube-api-access-7fr8z" pod "a7e743f3-5463-4ae1-9c8a-75a892fbaceb" (UID: "a7e743f3-5463-4ae1-9c8a-75a892fbaceb"). InnerVolumeSpecName "kube-api-access-7fr8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:22.696541 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.696524 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:46:22.791529 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.791500 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3575f09c-1832-468b-b938-1fd125084dce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3575f09c-1832-468b-b938-1fd125084dce-plugins-config-volume\") pod \"3575f09c-1832-468b-b938-1fd125084dce\" (UID: \"3575f09c-1832-468b-b938-1fd125084dce\") " Oct 30 22:46:22.791689 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.791546 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3575f09c-1832-468b-b938-1fd125084dce-kube-api-access-8f42v\" (UniqueName: \"kubernetes.io/projected/3575f09c-1832-468b-b938-1fd125084dce-kube-api-access-8f42v\") pod \"3575f09c-1832-468b-b938-1fd125084dce\" (UID: \"3575f09c-1832-468b-b938-1fd125084dce\") " Oct 30 22:46:22.791885 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.791864 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fr8z\" (UniqueName: \"kubernetes.io/projected/a7e743f3-5463-4ae1-9c8a-75a892fbaceb-kube-api-access-7fr8z\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:22.791964 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.791944 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3575f09c-1832-468b-b938-1fd125084dce-plugins-config-volume" pod "3575f09c-1832-468b-b938-1fd125084dce" (UID: "3575f09c-1832-468b-b938-1fd125084dce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:22.793685 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.793662 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3575f09c-1832-468b-b938-1fd125084dce-kube-api-access-8f42v" pod "3575f09c-1832-468b-b938-1fd125084dce" (UID: "3575f09c-1832-468b-b938-1fd125084dce"). InnerVolumeSpecName "kube-api-access-8f42v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:22.893047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.892950 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3575f09c-1832-468b-b938-1fd125084dce-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:22.893047 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:22.892985 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f42v\" (UniqueName: \"kubernetes.io/projected/3575f09c-1832-468b-b938-1fd125084dce-kube-api-access-8f42v\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:23.040868 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:23.040828 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf"] Oct 30 22:46:23.044969 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:23.044939 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-23f3d4-6b7474cb45-m9qzf"] Oct 30 22:46:23.641283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:23.641178 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" Oct 30 22:46:23.641283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:23.641234 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j" event={"ID":"3575f09c-1832-468b-b938-1fd125084dce","Type":"ContainerDied","Data":"78cae9b7b8a1fc61cbc8ae7b05b637102222a0849e9180339709f76006ff38a1"} Oct 30 22:46:23.641770 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:23.641294 5226 scope.go:122] "RemoveContainer" containerID="eae287d37ea1a123f1dc742c407176339be3debe822ef3a7339e8a0caefa9fac" Oct 30 22:46:23.796695 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:23.796666 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j"] Oct 30 22:46:23.803602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:23.803578 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23f3d4-564458bb8c-jq54j"] Oct 30 22:46:24.411877 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:24.411843 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:46:24.903612 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:24.903577 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3575f09c-1832-468b-b938-1fd125084dce" path="/var/lib/kubelet/pods/3575f09c-1832-468b-b938-1fd125084dce/volumes" Oct 30 22:46:24.903991 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:24.903973 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e743f3-5463-4ae1-9c8a-75a892fbaceb" path="/var/lib/kubelet/pods/a7e743f3-5463-4ae1-9c8a-75a892fbaceb/volumes" Oct 30 22:46:33.077671 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:33.077635 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc"] Oct 30 22:46:48.098215 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.098144 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" podUID="2f91d146-b3af-4895-87f4-011002cc075a" containerName="main" containerID="cri-o://53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9" gracePeriod=45 Oct 30 22:46:48.358234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.358164 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:46:48.462069 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462035 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f91d146-b3af-4895-87f4-011002cc075a-kube-api-access-pjbsr\" (UniqueName: \"kubernetes.io/projected/2f91d146-b3af-4895-87f4-011002cc075a-kube-api-access-pjbsr\") pod \"2f91d146-b3af-4895-87f4-011002cc075a\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " Oct 30 22:46:48.462273 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462091 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-dshm\") pod \"2f91d146-b3af-4895-87f4-011002cc075a\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " Oct 30 22:46:48.462273 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462147 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f91d146-b3af-4895-87f4-011002cc075a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91d146-b3af-4895-87f4-011002cc075a-tls-certs\") pod \"2f91d146-b3af-4895-87f4-011002cc075a\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " Oct 30 22:46:48.462273 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462222 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-home\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-home\") pod \"2f91d146-b3af-4895-87f4-011002cc075a\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " Oct 30 22:46:48.462273 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462245 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-tmp-dir\") pod \"2f91d146-b3af-4895-87f4-011002cc075a\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " Oct 30 22:46:48.462273 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462273 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-model-cache\") pod \"2f91d146-b3af-4895-87f4-011002cc075a\" (UID: \"2f91d146-b3af-4895-87f4-011002cc075a\") " Oct 30 22:46:48.462584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462551 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-home" pod "2f91d146-b3af-4895-87f4-011002cc075a" (UID: "2f91d146-b3af-4895-87f4-011002cc075a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:48.462584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462564 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-tmp-dir" pod "2f91d146-b3af-4895-87f4-011002cc075a" (UID: "2f91d146-b3af-4895-87f4-011002cc075a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:48.462752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462591 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-model-cache" pod "2f91d146-b3af-4895-87f4-011002cc075a" (UID: "2f91d146-b3af-4895-87f4-011002cc075a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:48.462752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462691 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:48.462752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462712 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:48.462752 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.462727 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:48.464349 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.464322 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f91d146-b3af-4895-87f4-011002cc075a-tls-certs" pod "2f91d146-b3af-4895-87f4-011002cc075a" (UID: "2f91d146-b3af-4895-87f4-011002cc075a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:48.464458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.464428 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-dshm" pod "2f91d146-b3af-4895-87f4-011002cc075a" (UID: "2f91d146-b3af-4895-87f4-011002cc075a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:46:48.464458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.464431 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f91d146-b3af-4895-87f4-011002cc075a-kube-api-access-pjbsr" pod "2f91d146-b3af-4895-87f4-011002cc075a" (UID: "2f91d146-b3af-4895-87f4-011002cc075a"). InnerVolumeSpecName "kube-api-access-pjbsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:48.563743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.563713 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjbsr\" (UniqueName: \"kubernetes.io/projected/2f91d146-b3af-4895-87f4-011002cc075a-kube-api-access-pjbsr\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:48.563743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.563741 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f91d146-b3af-4895-87f4-011002cc075a-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:48.563923 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.563750 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f91d146-b3af-4895-87f4-011002cc075a-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:48.778907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.778867 5226 generic.go:356] "Generic (PLEG): container finished" podID="2f91d146-b3af-4895-87f4-011002cc075a" containerID="53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9" exitCode=0 Oct 30 22:46:48.778907 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.778905 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" event={"ID":"2f91d146-b3af-4895-87f4-011002cc075a","Type":"ContainerDied","Data":"53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9"} Oct 30 22:46:48.779166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.778950 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" event={"ID":"2f91d146-b3af-4895-87f4-011002cc075a","Type":"ContainerDied","Data":"99a545667989caf17596eda9375adac01079428d4ea313159c176b38b74d1022"} Oct 30 22:46:48.779166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.778948 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc" Oct 30 22:46:48.779166 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.778965 5226 scope.go:122] "RemoveContainer" containerID="53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9" Oct 30 22:46:48.790924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.790897 5226 scope.go:122] "RemoveContainer" containerID="53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9" Oct 30 22:46:48.791249 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:48.791191 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9\": container with ID starting with 53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9 not found: ID does not exist" containerID="53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9" Oct 30 22:46:48.791346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.791261 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9"} err="failed to get container status \"53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9\": rpc error: code = NotFound desc = could not find container \"53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9\": container with ID starting with 53e9bcbae7425d02b9b4af7318d66b53302b5b109614659d93f80510913397e9 not found: ID does not exist" Oct 30 22:46:48.936098 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.936064 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc"] Oct 30 22:46:48.983742 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:48.983707 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fb3406/test-model-fb3406-kserve-866d67f669-bcsxc"] Oct 30 22:46:49.407130 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:49.407103 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:46:49.421513 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:49.421491 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:46:50.904820 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:46:50.904784 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f91d146-b3af-4895-87f4-011002cc075a" path="/var/lib/kubelet/pods/2f91d146-b3af-4895-87f4-011002cc075a/volumes" Oct 30 22:46:54.412036 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:46:54.411996 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:47:05.048792 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:05.048707 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:47:17.440947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:17.440907 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5"] Oct 30 22:47:24.411568 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:47:24.411527 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:47:31.613497 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:31.613466 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:47:32.333099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.333061 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26"] Oct 30 22:47:32.333337 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.333287 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" podUID="9789bc21-7597-4d25-b8a2-43c22c6fe11d" containerName="maas-api" containerID="cri-o://99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c" gracePeriod=30 Oct 30 22:47:32.461620 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.461577 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" containerID="cri-o://9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39" gracePeriod=45 Oct 30 22:47:32.631805 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.631768 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:47:32.711432 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.711397 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9789bc21-7597-4d25-b8a2-43c22c6fe11d-kube-api-access-852bp\" (UniqueName: \"kubernetes.io/projected/9789bc21-7597-4d25-b8a2-43c22c6fe11d-kube-api-access-852bp\") pod \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " Oct 30 22:47:32.711644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.711444 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls\") pod \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " Oct 30 22:47:32.711644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.711464 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9789bc21-7597-4d25-b8a2-43c22c6fe11d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9789bc21-7597-4d25-b8a2-43c22c6fe11d-openshift-service-ca\") pod \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " Oct 30 22:47:32.711644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.711529 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-metrics-certs\") pod \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\" (UID: \"9789bc21-7597-4d25-b8a2-43c22c6fe11d\") " Oct 30 22:47:32.712458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.712388 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9789bc21-7597-4d25-b8a2-43c22c6fe11d-openshift-service-ca" pod "9789bc21-7597-4d25-b8a2-43c22c6fe11d" (UID: "9789bc21-7597-4d25-b8a2-43c22c6fe11d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:32.714471 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.714414 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls" pod "9789bc21-7597-4d25-b8a2-43c22c6fe11d" (UID: "9789bc21-7597-4d25-b8a2-43c22c6fe11d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:32.714729 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.714695 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-metrics-certs" pod "9789bc21-7597-4d25-b8a2-43c22c6fe11d" (UID: "9789bc21-7597-4d25-b8a2-43c22c6fe11d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:32.714828 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.714788 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9789bc21-7597-4d25-b8a2-43c22c6fe11d-kube-api-access-852bp" pod "9789bc21-7597-4d25-b8a2-43c22c6fe11d" (UID: "9789bc21-7597-4d25-b8a2-43c22c6fe11d"). InnerVolumeSpecName "kube-api-access-852bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:32.736224 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.736179 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:47:32.812849 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.812808 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-model-cache\") pod \"15178a87-7ac5-405e-9b46-1a8de8861922\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " Oct 30 22:47:32.813015 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.812903 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-home\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-home\") pod \"15178a87-7ac5-405e-9b46-1a8de8861922\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " Oct 30 22:47:32.813015 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.812996 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-tmp-dir\") pod \"15178a87-7ac5-405e-9b46-1a8de8861922\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " Oct 30 22:47:32.813120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813029 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-dshm\") pod \"15178a87-7ac5-405e-9b46-1a8de8861922\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " Oct 30 22:47:32.813120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813089 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-model-cache" pod "15178a87-7ac5-405e-9b46-1a8de8861922" (UID: "15178a87-7ac5-405e-9b46-1a8de8861922"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:32.813120 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813103 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15178a87-7ac5-405e-9b46-1a8de8861922-kube-api-access-jtg47\" (UniqueName: \"kubernetes.io/projected/15178a87-7ac5-405e-9b46-1a8de8861922-kube-api-access-jtg47\") pod \"15178a87-7ac5-405e-9b46-1a8de8861922\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " Oct 30 22:47:32.813290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813150 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15178a87-7ac5-405e-9b46-1a8de8861922-tls-certs\" (UniqueName: \"kubernetes.io/secret/15178a87-7ac5-405e-9b46-1a8de8861922-tls-certs\") pod \"15178a87-7ac5-405e-9b46-1a8de8861922\" (UID: \"15178a87-7ac5-405e-9b46-1a8de8861922\") " Oct 30 22:47:32.813290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813262 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-home" pod "15178a87-7ac5-405e-9b46-1a8de8861922" (UID: "15178a87-7ac5-405e-9b46-1a8de8861922"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:32.813368 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813304 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-tmp-dir" pod "15178a87-7ac5-405e-9b46-1a8de8861922" (UID: "15178a87-7ac5-405e-9b46-1a8de8861922"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:32.813589 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813568 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-852bp\" (UniqueName: \"kubernetes.io/projected/9789bc21-7597-4d25-b8a2-43c22c6fe11d-kube-api-access-852bp\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.813650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813595 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.813650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813612 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.813650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813626 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9789bc21-7597-4d25-b8a2-43c22c6fe11d-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.813650 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813639 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9789bc21-7597-4d25-b8a2-43c22c6fe11d-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.813804 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813652 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.813804 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.813665 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.815442 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.815419 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-dshm" pod "15178a87-7ac5-405e-9b46-1a8de8861922" (UID: "15178a87-7ac5-405e-9b46-1a8de8861922"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:32.815537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.815450 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15178a87-7ac5-405e-9b46-1a8de8861922-tls-certs" pod "15178a87-7ac5-405e-9b46-1a8de8861922" (UID: "15178a87-7ac5-405e-9b46-1a8de8861922"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:32.815537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.815472 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15178a87-7ac5-405e-9b46-1a8de8861922-kube-api-access-jtg47" pod "15178a87-7ac5-405e-9b46-1a8de8861922" (UID: "15178a87-7ac5-405e-9b46-1a8de8861922"). InnerVolumeSpecName "kube-api-access-jtg47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:32.915393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.915354 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtg47\" (UniqueName: \"kubernetes.io/projected/15178a87-7ac5-405e-9b46-1a8de8861922-kube-api-access-jtg47\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.915393 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.915381 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15178a87-7ac5-405e-9b46-1a8de8861922-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:32.915533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:32.915395 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15178a87-7ac5-405e-9b46-1a8de8861922-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:33.015538 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.015503 5226 generic.go:356] "Generic (PLEG): container finished" podID="9789bc21-7597-4d25-b8a2-43c22c6fe11d" containerID="99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c" exitCode=0 Oct 30 22:47:33.015750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.015558 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" Oct 30 22:47:33.015750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.015601 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" event={"ID":"9789bc21-7597-4d25-b8a2-43c22c6fe11d","Type":"ContainerDied","Data":"99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c"} Oct 30 22:47:33.015750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.015664 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26" event={"ID":"9789bc21-7597-4d25-b8a2-43c22c6fe11d","Type":"ContainerDied","Data":"4ae5bf1b202333746d93ecd1f8428cb02b17fb951e624470691905bec62d452d"} Oct 30 22:47:33.015750 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.015693 5226 scope.go:122] "RemoveContainer" containerID="99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c" Oct 30 22:47:33.017717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.017214 5226 generic.go:356] "Generic (PLEG): container finished" podID="15178a87-7ac5-405e-9b46-1a8de8861922" containerID="9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39" exitCode=0 Oct 30 22:47:33.017717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.017279 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" Oct 30 22:47:33.017717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.017233 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" event={"ID":"15178a87-7ac5-405e-9b46-1a8de8861922","Type":"ContainerDied","Data":"9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39"} Oct 30 22:47:33.017717 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.017351 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5" event={"ID":"15178a87-7ac5-405e-9b46-1a8de8861922","Type":"ContainerDied","Data":"9902d914543f1b13b56f84d83efbe25d1f25f98d4948d1ad50f8550d6a66a902"} Oct 30 22:47:33.027403 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.027354 5226 scope.go:122] "RemoveContainer" containerID="99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c" Oct 30 22:47:33.027717 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:47:33.027698 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c\": container with ID starting with 99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c not found: ID does not exist" containerID="99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c" Oct 30 22:47:33.027800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.027730 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c"} err="failed to get container status \"99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c\": rpc error: code = NotFound desc = could not find container \"99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c\": container with ID starting with 99012f4074a6ce3589814cc44a072e1ea34f3d0f2002d686b706dd55f1ebd67c not found: ID does not exist" Oct 30 22:47:33.027800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.027755 5226 scope.go:122] "RemoveContainer" containerID="9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39" Oct 30 22:47:33.036488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.036463 5226 scope.go:122] "RemoveContainer" containerID="9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39" Oct 30 22:47:33.036834 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:47:33.036814 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39\": container with ID starting with 9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39 not found: ID does not exist" containerID="9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39" Oct 30 22:47:33.036902 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.036843 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39"} err="failed to get container status \"9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39\": rpc error: code = NotFound desc = could not find container \"9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39\": container with ID starting with 9d75216288e901287ae9ed76c294d4337964edd87ddaa14856d79a05895f9e39 not found: ID does not exist" Oct 30 22:47:33.175821 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.175789 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26"] Oct 30 22:47:33.185425 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.185386 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fb3406-7896c4b6b9-v6m26"] Oct 30 22:47:33.293527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.293496 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5"] Oct 30 22:47:33.295382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:33.295353 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b13755/simulator-w4-b13755-kserve-5c558499c6-s29m5"] Oct 30 22:47:34.904198 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:34.904160 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" path="/var/lib/kubelet/pods/15178a87-7ac5-405e-9b46-1a8de8861922/volumes" Oct 30 22:47:34.904639 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:34.904623 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9789bc21-7597-4d25-b8a2-43c22c6fe11d" path="/var/lib/kubelet/pods/9789bc21-7597-4d25-b8a2-43c22c6fe11d/volumes" Oct 30 22:47:41.751494 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:41.751452 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq"] Oct 30 22:47:41.751963 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:41.751689 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" containerID="cri-o://c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464" gracePeriod=30 Oct 30 22:47:46.309989 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:46.309957 5226 patch_prober.go:29] interesting pod/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:47:46.310375 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:46.310021 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:47:47.007654 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.007631 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:47:47.089398 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.089313 5226 generic.go:356] "Generic (PLEG): container finished" podID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerID="c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464" exitCode=0 Oct 30 22:47:47.089398 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.089375 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" Oct 30 22:47:47.089577 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.089402 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" event={"ID":"50cebe87-7505-4ac6-af29-b423530a1ad3","Type":"ContainerDied","Data":"c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464"} Oct 30 22:47:47.089577 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.089479 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq" event={"ID":"50cebe87-7505-4ac6-af29-b423530a1ad3","Type":"ContainerDied","Data":"aa646f7701962adfa9e34647b63000554ee27a6a731bcbaf0c16db86ec02f924"} Oct 30 22:47:47.089577 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.089501 5226 scope.go:122] "RemoveContainer" containerID="c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464" Oct 30 22:47:47.098212 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.098184 5226 scope.go:122] "RemoveContainer" containerID="a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be" Oct 30 22:47:47.106485 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.106469 5226 scope.go:122] "RemoveContainer" containerID="c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464" Oct 30 22:47:47.106752 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:47:47.106731 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464\": container with ID starting with c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464 not found: ID does not exist" containerID="c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464" Oct 30 22:47:47.106823 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.106761 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464"} err="failed to get container status \"c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464\": rpc error: code = NotFound desc = could not find container \"c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464\": container with ID starting with c4f1f56041eaaeed70ee5c49bdb347a2b9bf672fdef16bd0d5a56481c1330464 not found: ID does not exist" Oct 30 22:47:47.106823 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.106780 5226 scope.go:122] "RemoveContainer" containerID="a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be" Oct 30 22:47:47.107020 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:47:47.107004 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be\": container with ID starting with a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be not found: ID does not exist" containerID="a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be" Oct 30 22:47:47.107061 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.107026 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be"} err="failed to get container status \"a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be\": rpc error: code = NotFound desc = could not find container \"a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be\": container with ID starting with a3e660fd253639dd43f208a6cf9912940a77b65aa2477d580910f08cd4c658be not found: ID does not exist" Oct 30 22:47:47.160505 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160468 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-socket\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.160700 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160566 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-podinfo\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.160700 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160624 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-credential-socket\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.160700 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160651 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-certs\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.160886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160715 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-token\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-token\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.160886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160755 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-data\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.160886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160781 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cebe87-7505-4ac6-af29-b423530a1ad3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cebe87-7505-4ac6-af29-b423530a1ad3-istiod-ca-cert\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.160886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160800 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-socket" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.160886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160808 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-envoy\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.160886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.160881 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-kube-api-access-2dqqh\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-kube-api-access-2dqqh\") pod \"50cebe87-7505-4ac6-af29-b423530a1ad3\" (UID: \"50cebe87-7505-4ac6-af29-b423530a1ad3\") " Oct 30 22:47:47.161192 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.161000 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-certs" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.161301 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.161280 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-credential-socket" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.161368 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.161215 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cebe87-7505-4ac6-af29-b423530a1ad3-istiod-ca-cert" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:47.161441 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.161420 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.161492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.161448 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.161492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.161462 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.161492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.161475 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cebe87-7505-4ac6-af29-b423530a1ad3-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.161492 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.161471 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-data" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.163297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.163269 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-token" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:47.163297 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.163276 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-envoy" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.163458 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.163367 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-podinfo" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:47.163602 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.163583 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-kube-api-access-2dqqh" pod "50cebe87-7505-4ac6-af29-b423530a1ad3" (UID: "50cebe87-7505-4ac6-af29-b423530a1ad3"). InnerVolumeSpecName "kube-api-access-2dqqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:47.262553 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.262520 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.262553 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.262552 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dqqh\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-kube-api-access-2dqqh\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.262553 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.262567 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.262878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.262580 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.262878 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.262591 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cebe87-7505-4ac6-af29-b423530a1ad3-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.504156 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.504123 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq"] Oct 30 22:47:47.506542 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:47.506523 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fb3406-openshift-default-75d976cfdf-rgthq"] Oct 30 22:47:48.904814 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:48.904770 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" path="/var/lib/kubelet/pods/50cebe87-7505-4ac6-af29-b423530a1ad3/volumes" Oct 30 22:47:53.222540 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:53.222508 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:47:53.961226 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:53.961181 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg"] Oct 30 22:47:53.961477 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:53.961443 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" containerID="cri-o://a04ebb720c83e37e507027d5e4a26419b6baf5aff3243e4d3986c0e42d11f272" gracePeriod=30 Oct 30 22:47:54.007175 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:54.007138 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw"] Oct 30 22:47:54.119778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:54.119745 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646"] Oct 30 22:47:54.410805 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:47:54.410724 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:47:57.382929 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.382893 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv"] Oct 30 22:47:57.383427 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.383091 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" podUID="e1cfb0fc-ba6f-4133-94db-01f8c7fed971" containerName="maas-api" containerID="cri-o://87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc" gracePeriod=30 Oct 30 22:47:57.672248 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.672220 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:47:57.772066 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.772028 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-kube-api-access-9pgtq\" (UniqueName: \"kubernetes.io/projected/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-kube-api-access-9pgtq\") pod \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " Oct 30 22:47:57.772283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.772128 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-metrics-certs\") pod \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " Oct 30 22:47:57.772283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.772233 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-openshift-service-ca\") pod \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " Oct 30 22:47:57.772409 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.772317 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls\") pod \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\" (UID: \"e1cfb0fc-ba6f-4133-94db-01f8c7fed971\") " Oct 30 22:47:57.772622 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.772590 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-openshift-service-ca" pod "e1cfb0fc-ba6f-4133-94db-01f8c7fed971" (UID: "e1cfb0fc-ba6f-4133-94db-01f8c7fed971"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:57.772756 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.772727 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:57.774242 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.774194 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls" pod "e1cfb0fc-ba6f-4133-94db-01f8c7fed971" (UID: "e1cfb0fc-ba6f-4133-94db-01f8c7fed971"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:57.774358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.774253 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-kube-api-access-9pgtq" pod "e1cfb0fc-ba6f-4133-94db-01f8c7fed971" (UID: "e1cfb0fc-ba6f-4133-94db-01f8c7fed971"). InnerVolumeSpecName "kube-api-access-9pgtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:57.774358 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.774311 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-metrics-certs" pod "e1cfb0fc-ba6f-4133-94db-01f8c7fed971" (UID: "e1cfb0fc-ba6f-4133-94db-01f8c7fed971"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:57.874191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.874157 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:57.874191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.874185 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:57.874191 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:57.874196 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pgtq\" (UniqueName: \"kubernetes.io/projected/e1cfb0fc-ba6f-4133-94db-01f8c7fed971-kube-api-access-9pgtq\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:58.146957 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.146920 5226 generic.go:356] "Generic (PLEG): container finished" podID="e1cfb0fc-ba6f-4133-94db-01f8c7fed971" containerID="87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc" exitCode=0 Oct 30 22:47:58.147121 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.146981 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" Oct 30 22:47:58.147121 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.147005 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" event={"ID":"e1cfb0fc-ba6f-4133-94db-01f8c7fed971","Type":"ContainerDied","Data":"87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc"} Oct 30 22:47:58.147121 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.147060 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv" event={"ID":"e1cfb0fc-ba6f-4133-94db-01f8c7fed971","Type":"ContainerDied","Data":"0183884a89817a0fc1dc1a261792c66ad5ca5cf9dc825a3d6da8aa9488e10ac4"} Oct 30 22:47:58.147121 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.147081 5226 scope.go:122] "RemoveContainer" containerID="87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc" Oct 30 22:47:58.157141 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.157124 5226 scope.go:122] "RemoveContainer" containerID="87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc" Oct 30 22:47:58.157415 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:47:58.157398 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc\": container with ID starting with 87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc not found: ID does not exist" containerID="87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc" Oct 30 22:47:58.157506 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.157423 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc"} err="failed to get container status \"87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc\": rpc error: code = NotFound desc = could not find container \"87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc\": container with ID starting with 87d0d5c60bda9f10f14b19548540c8ba020bd8274378ba0409d17b2cf73c0cdc not found: ID does not exist" Oct 30 22:47:58.300408 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.300373 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv"] Oct 30 22:47:58.346528 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.346496 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b13755-57c78bc5cb-qxcrv"] Oct 30 22:47:58.904560 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:58.904526 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1cfb0fc-ba6f-4133-94db-01f8c7fed971" path="/var/lib/kubelet/pods/e1cfb0fc-ba6f-4133-94db-01f8c7fed971/volumes" Oct 30 22:47:59.008354 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.008321 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" podUID="09534bf8-e3e6-420e-8172-631254b050cd" containerName="payload-processing" containerID="cri-o://1b168d7bb26c93eadb8cd234997f0589cb8a615951107d7ed52bfe19ca65f059" gracePeriod=25 Oct 30 22:47:59.120833 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.120771 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" podUID="e3eebb59-0f23-4b2d-b862-1a1909cc4e57" containerName="payload-pre-processing" containerID="cri-o://747c23cd3ed57b2466f2487b2133b4cda99a03609825f2f369a23934f1509a12" gracePeriod=25 Oct 30 22:47:59.153189 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.153158 5226 generic.go:356] "Generic (PLEG): container finished" podID="81d68e5b-715f-4393-a12a-f06477f82928" containerID="a04ebb720c83e37e507027d5e4a26419b6baf5aff3243e4d3986c0e42d11f272" exitCode=0 Oct 30 22:47:59.153340 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.153239 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" event={"ID":"81d68e5b-715f-4393-a12a-f06477f82928","Type":"ContainerDied","Data":"a04ebb720c83e37e507027d5e4a26419b6baf5aff3243e4d3986c0e42d11f272"} Oct 30 22:47:59.225970 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.225945 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:47:59.391595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391556 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-envoy\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.391802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391659 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-istio-token\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-istio-token\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.391802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391706 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-credential-socket\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.391802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391744 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-certs\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.391977 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391831 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d68e5b-715f-4393-a12a-f06477f82928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d68e5b-715f-4393-a12a-f06477f82928-istiod-ca-cert\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.391977 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391866 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81d68e5b-715f-4393-a12a-f06477f82928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d68e5b-715f-4393-a12a-f06477f82928-istio-podinfo\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.391977 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391905 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-kube-api-access-mnqlm\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-kube-api-access-mnqlm\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.391977 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391942 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-socket\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.392183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.391981 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-data\") pod \"81d68e5b-715f-4393-a12a-f06477f82928\" (UID: \"81d68e5b-715f-4393-a12a-f06477f82928\") " Oct 30 22:47:59.392183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.392066 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-certs" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.392183 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.392081 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-credential-socket" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.392385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.392305 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-socket" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.392385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.392347 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-data" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.392385 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.392371 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d68e5b-715f-4393-a12a-f06477f82928-istiod-ca-cert" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:59.392526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.392454 5226 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-credential-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.392526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.392479 5226 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.392526 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.392496 5226 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-workload-socket\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.394084 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.394048 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-kube-api-access-mnqlm" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "kube-api-access-mnqlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.394312 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.394158 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81d68e5b-715f-4393-a12a-f06477f82928-istio-podinfo" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:59.394312 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.394266 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-envoy" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:59.394411 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.394350 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-istio-token" pod "81d68e5b-715f-4393-a12a-f06477f82928" (UID: "81d68e5b-715f-4393-a12a-f06477f82928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:59.493565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.493478 5226 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-istio-token\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.493565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.493510 5226 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81d68e5b-715f-4393-a12a-f06477f82928-istiod-ca-cert\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.493565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.493520 5226 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81d68e5b-715f-4393-a12a-f06477f82928-istio-podinfo\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.493565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.493531 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnqlm\" (UniqueName: \"kubernetes.io/projected/81d68e5b-715f-4393-a12a-f06477f82928-kube-api-access-mnqlm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.493565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.493541 5226 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-data\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:59.493565 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:47:59.493550 5226 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81d68e5b-715f-4393-a12a-f06477f82928-istio-envoy\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:00.160190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:00.160157 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" event={"ID":"81d68e5b-715f-4393-a12a-f06477f82928","Type":"ContainerDied","Data":"ff8f960470de3e17d2024b7e005f3684d47372801be2297c4faddcf4bde6a0b8"} Oct 30 22:48:00.160190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:00.160177 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg" Oct 30 22:48:00.160675 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:00.160241 5226 scope.go:122] "RemoveContainer" containerID="a04ebb720c83e37e507027d5e4a26419b6baf5aff3243e4d3986c0e42d11f272" Oct 30 22:48:00.317687 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:00.317655 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg"] Oct 30 22:48:00.363511 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:00.363478 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-m55sg"] Oct 30 22:48:00.904662 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:00.904627 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d68e5b-715f-4393-a12a-f06477f82928" path="/var/lib/kubelet/pods/81d68e5b-715f-4393-a12a-f06477f82928/volumes" Oct 30 22:48:01.542619 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:01.542587 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" podUID="e3eebb59-0f23-4b2d-b862-1a1909cc4e57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:48:01.543064 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:01.542637 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" podUID="09534bf8-e3e6-420e-8172-631254b050cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:48:11.542069 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:11.542024 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" podUID="e3eebb59-0f23-4b2d-b862-1a1909cc4e57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:48:11.542069 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:11.542047 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" podUID="09534bf8-e3e6-420e-8172-631254b050cd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:48:16.677605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:16.677575 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:48:19.255056 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.255015 5226 generic.go:356] "Generic (PLEG): container finished" podID="09534bf8-e3e6-420e-8172-631254b050cd" containerID="1b168d7bb26c93eadb8cd234997f0589cb8a615951107d7ed52bfe19ca65f059" exitCode=0 Oct 30 22:48:19.255499 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.255117 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" event={"ID":"09534bf8-e3e6-420e-8172-631254b050cd","Type":"ContainerDied","Data":"1b168d7bb26c93eadb8cd234997f0589cb8a615951107d7ed52bfe19ca65f059"} Oct 30 22:48:19.257364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.257262 5226 generic.go:356] "Generic (PLEG): container finished" podID="e3eebb59-0f23-4b2d-b862-1a1909cc4e57" containerID="747c23cd3ed57b2466f2487b2133b4cda99a03609825f2f369a23934f1509a12" exitCode=0 Oct 30 22:48:19.257364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.257288 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" event={"ID":"e3eebb59-0f23-4b2d-b862-1a1909cc4e57","Type":"ContainerDied","Data":"747c23cd3ed57b2466f2487b2133b4cda99a03609825f2f369a23934f1509a12"} Oct 30 22:48:19.278089 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.278032 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:48:19.385960 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.385931 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09534bf8-e3e6-420e-8172-631254b050cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09534bf8-e3e6-420e-8172-631254b050cd-plugins-config-volume\") pod \"09534bf8-e3e6-420e-8172-631254b050cd\" (UID: \"09534bf8-e3e6-420e-8172-631254b050cd\") " Oct 30 22:48:19.386096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.385959 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:48:19.386096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.386086 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09534bf8-e3e6-420e-8172-631254b050cd-kube-api-access-cvl7c\" (UniqueName: \"kubernetes.io/projected/09534bf8-e3e6-420e-8172-631254b050cd-kube-api-access-cvl7c\") pod \"09534bf8-e3e6-420e-8172-631254b050cd\" (UID: \"09534bf8-e3e6-420e-8172-631254b050cd\") " Oct 30 22:48:19.386290 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.386269 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09534bf8-e3e6-420e-8172-631254b050cd-plugins-config-volume" pod "09534bf8-e3e6-420e-8172-631254b050cd" (UID: "09534bf8-e3e6-420e-8172-631254b050cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:19.388199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.388174 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09534bf8-e3e6-420e-8172-631254b050cd-kube-api-access-cvl7c" pod "09534bf8-e3e6-420e-8172-631254b050cd" (UID: "09534bf8-e3e6-420e-8172-631254b050cd"). InnerVolumeSpecName "kube-api-access-cvl7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:19.486716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.486686 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-plugins-config-volume\") pod \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\" (UID: \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\") " Oct 30 22:48:19.486904 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.486827 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-kube-api-access-9l9b8\" (UniqueName: \"kubernetes.io/projected/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-kube-api-access-9l9b8\") pod \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\" (UID: \"e3eebb59-0f23-4b2d-b862-1a1909cc4e57\") " Oct 30 22:48:19.487058 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.487036 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-plugins-config-volume" pod "e3eebb59-0f23-4b2d-b862-1a1909cc4e57" (UID: "e3eebb59-0f23-4b2d-b862-1a1909cc4e57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:19.487155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.487137 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:19.487230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.487161 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvl7c\" (UniqueName: \"kubernetes.io/projected/09534bf8-e3e6-420e-8172-631254b050cd-kube-api-access-cvl7c\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:19.487230 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.487178 5226 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09534bf8-e3e6-420e-8172-631254b050cd-plugins-config-volume\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:19.488927 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.488906 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-kube-api-access-9l9b8" pod "e3eebb59-0f23-4b2d-b862-1a1909cc4e57" (UID: "e3eebb59-0f23-4b2d-b862-1a1909cc4e57"). InnerVolumeSpecName "kube-api-access-9l9b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:19.588457 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:19.588378 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9b8\" (UniqueName: \"kubernetes.io/projected/e3eebb59-0f23-4b2d-b862-1a1909cc4e57-kube-api-access-9l9b8\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:20.265264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.265233 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" Oct 30 22:48:20.265264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.265252 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646" event={"ID":"e3eebb59-0f23-4b2d-b862-1a1909cc4e57","Type":"ContainerDied","Data":"d7d9bdfa46159e5f9409abc42c2a22da4b24f0f08f249be58dc71340132ee14c"} Oct 30 22:48:20.265785 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.265324 5226 scope.go:122] "RemoveContainer" containerID="747c23cd3ed57b2466f2487b2133b4cda99a03609825f2f369a23934f1509a12" Oct 30 22:48:20.266743 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.266717 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" event={"ID":"09534bf8-e3e6-420e-8172-631254b050cd","Type":"ContainerDied","Data":"10ed827468839e85b6113fca7e3c3bac7da82628c5e70d13fdb0b3eb2c4d6185"} Oct 30 22:48:20.266843 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.266744 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw" Oct 30 22:48:20.274488 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.274450 5226 scope.go:122] "RemoveContainer" containerID="1b168d7bb26c93eadb8cd234997f0589cb8a615951107d7ed52bfe19ca65f059" Oct 30 22:48:20.418605 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.418577 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646"] Oct 30 22:48:20.464932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.464905 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-afb98c-76498858ff-f4646"] Oct 30 22:48:20.567322 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.567255 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw"] Oct 30 22:48:20.571656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.571632 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-afb98c-fd7dd4465-wd8dw"] Oct 30 22:48:20.904025 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.903947 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09534bf8-e3e6-420e-8172-631254b050cd" path="/var/lib/kubelet/pods/09534bf8-e3e6-420e-8172-631254b050cd/volumes" Oct 30 22:48:20.904372 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:20.904358 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3eebb59-0f23-4b2d-b862-1a1909cc4e57" path="/var/lib/kubelet/pods/e3eebb59-0f23-4b2d-b862-1a1909cc4e57/volumes" Oct 30 22:48:24.409555 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:48:24.409522 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:48:42.812395 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:48:42.812359 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:48:54.410098 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:48:54.410058 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:49:11.336778 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:11.336735 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:49:20.278896 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:20.278864 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q"] Oct 30 22:49:20.324831 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:20.324796 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c"] Oct 30 22:49:24.410111 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:49:24.410069 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:49:35.298282 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.298218 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" containerID="cri-o://d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac" gracePeriod=45 Oct 30 22:49:35.343848 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.343796 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" containerID="cri-o://27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c" gracePeriod=45 Oct 30 22:49:35.346190 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346165 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r"] Oct 30 22:49:35.346679 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346657 5226 cpu_manager.go:414] "containerMap: removing container" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346680 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346701 5226 cpu_manager.go:414] "containerMap: removing container" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346709 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346725 5226 cpu_manager.go:414] "containerMap: removing container" podUID="2f91d146-b3af-4895-87f4-011002cc075a" containerName="main" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346732 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f91d146-b3af-4895-87f4-011002cc075a" containerName="main" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346741 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346749 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346760 5226 cpu_manager.go:414] "containerMap: removing container" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346767 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346781 5226 cpu_manager.go:414] "containerMap: removing container" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" Oct 30 22:49:35.346786 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346788 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346796 5226 cpu_manager.go:414] "containerMap: removing container" podUID="94109748-986f-4293-ad0a-8795650676e9" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346804 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94109748-986f-4293-ad0a-8795650676e9" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346816 5226 cpu_manager.go:414] "containerMap: removing container" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346823 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346833 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346839 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346848 5226 cpu_manager.go:414] "containerMap: removing container" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346854 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346864 5226 cpu_manager.go:414] "containerMap: removing container" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346871 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346887 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ee04e125-52ff-47e2-b35d-94abe86c6ed1" containerName="payload-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346894 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee04e125-52ff-47e2-b35d-94abe86c6ed1" containerName="payload-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346905 5226 cpu_manager.go:414] "containerMap: removing container" podUID="7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346911 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346926 5226 cpu_manager.go:414] "containerMap: removing container" podUID="e1cfb0fc-ba6f-4133-94db-01f8c7fed971" containerName="maas-api" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346933 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1cfb0fc-ba6f-4133-94db-01f8c7fed971" containerName="maas-api" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346943 5226 cpu_manager.go:414] "containerMap: removing container" podUID="9789bc21-7597-4d25-b8a2-43c22c6fe11d" containerName="maas-api" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346950 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9789bc21-7597-4d25-b8a2-43c22c6fe11d" containerName="maas-api" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346969 5226 cpu_manager.go:414] "containerMap: removing container" podUID="3575f09c-1832-468b-b938-1fd125084dce" containerName="payload-pre-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346979 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3575f09c-1832-468b-b938-1fd125084dce" containerName="payload-pre-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346989 5226 cpu_manager.go:414] "containerMap: removing container" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.346996 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347007 5226 cpu_manager.go:414] "containerMap: removing container" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347014 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347028 5226 cpu_manager.go:414] "containerMap: removing container" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347036 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347047 5226 cpu_manager.go:414] "containerMap: removing container" podUID="a7e743f3-5463-4ae1-9c8a-75a892fbaceb" containerName="payload-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347055 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e743f3-5463-4ae1-9c8a-75a892fbaceb" containerName="payload-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347065 5226 cpu_manager.go:414] "containerMap: removing container" podUID="e3eebb59-0f23-4b2d-b862-1a1909cc4e57" containerName="payload-pre-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347072 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3eebb59-0f23-4b2d-b862-1a1909cc4e57" containerName="payload-pre-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347083 5226 cpu_manager.go:414] "containerMap: removing container" podUID="09534bf8-e3e6-420e-8172-631254b050cd" containerName="payload-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347091 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09534bf8-e3e6-420e-8172-631254b050cd" containerName="payload-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347103 5226 cpu_manager.go:414] "containerMap: removing container" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347110 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347121 5226 cpu_manager.go:414] "containerMap: removing container" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347129 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347147 5226 cpu_manager.go:414] "containerMap: removing container" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347154 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347164 5226 cpu_manager.go:414] "containerMap: removing container" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347171 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347184 5226 cpu_manager.go:414] "containerMap: removing container" podUID="7e772e6b-7473-4cd2-b906-efeca3dc1842" containerName="maas-api" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347191 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e772e6b-7473-4cd2-b906-efeca3dc1842" containerName="maas-api" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347219 5226 cpu_manager.go:414] "containerMap: removing container" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347227 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347238 5226 cpu_manager.go:414] "containerMap: removing container" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347245 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347257 5226 cpu_manager.go:414] "containerMap: removing container" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347266 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347366 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3eebb59-0f23-4b2d-b862-1a1909cc4e57" containerName="payload-pre-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347379 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="09534bf8-e3e6-420e-8172-631254b050cd" containerName="payload-processing" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347389 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="515e746e-b44c-463a-bc1a-5e57dfb5b920" containerName="main" Oct 30 22:49:35.347373 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347399 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347409 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347419 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347428 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e772e6b-7473-4cd2-b906-efeca3dc1842" containerName="maas-api" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347436 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5c60fa-3c6b-408c-84f5-b4d070aa8fb0" containerName="main" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347445 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347453 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347461 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="18303bc1-aa75-4a32-9173-a546eced5ba9" containerName="main" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347470 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="7054201d-3c4a-4839-b282-295d853eb43c" containerName="istio-proxy" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347480 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f91d146-b3af-4895-87f4-011002cc075a" containerName="main" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347489 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347496 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347505 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d68e5b-715f-4393-a12a-f06477f82928" containerName="istio-proxy" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347514 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="94109748-986f-4293-ad0a-8795650676e9" containerName="main" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347522 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347530 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6984d6-ca74-4760-9038-6a0d1faa77df" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347541 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="15178a87-7ac5-405e-9b46-1a8de8861922" containerName="main" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347548 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347557 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee04e125-52ff-47e2-b35d-94abe86c6ed1" containerName="payload-processing" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347566 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6b4c2c-a1f7-41f0-9705-10b4d666a11a" containerName="main" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347575 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1cfb0fc-ba6f-4133-94db-01f8c7fed971" containerName="maas-api" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347582 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="9789bc21-7597-4d25-b8a2-43c22c6fe11d" containerName="maas-api" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347593 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="3575f09c-1832-468b-b938-1fd125084dce" containerName="payload-pre-processing" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347603 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347611 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16c9ead-eb60-477e-b415-5a378862ffb5" containerName="istio-proxy" Oct 30 22:49:35.349603 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.347620 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e743f3-5463-4ae1-9c8a-75a892fbaceb" containerName="payload-processing" Oct 30 22:49:35.350889 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.350871 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.405500 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.405464 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rrw\" (UniqueName: \"kubernetes.io/projected/24dde2f5-964f-4779-bafe-213551f5e45b-kube-api-access-f8rrw\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.405657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.405515 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.405657 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.405576 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.412473 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.412451 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r"] Oct 30 22:49:35.463167 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.463138 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:49:35.463595 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.463574 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:49:35.504767 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.504047 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sq4d9\"" Oct 30 22:49:35.504767 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.504118 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:49:35.508235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.506868 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rrw\" (UniqueName: \"kubernetes.io/projected/24dde2f5-964f-4779-bafe-213551f5e45b-kube-api-access-f8rrw\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.508235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.506936 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.508235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.506983 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.508235 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.507876 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.511021 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.510973 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.567002 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.566970 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rrw\" (UniqueName: \"kubernetes.io/projected/24dde2f5-964f-4779-bafe-213551f5e45b-kube-api-access-f8rrw\") pod \"maas-api-revoke-sub-b23580df-2dd8r\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.589184 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.589157 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:49:35.599437 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.599419 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:49:35.634581 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.634547 5226 generic.go:356] "Generic (PLEG): container finished" podID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerID="27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c" exitCode=0 Oct 30 22:49:35.634716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.634598 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" Oct 30 22:49:35.634716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.634630 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" event={"ID":"38d3c91d-e41a-4a37-8367-53540a9f98b2","Type":"ContainerDied","Data":"27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c"} Oct 30 22:49:35.634716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.634675 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c" event={"ID":"38d3c91d-e41a-4a37-8367-53540a9f98b2","Type":"ContainerDied","Data":"da779fae194887775902792169630b7a35cf7edfc92e03196c5f5a5e33efa842"} Oct 30 22:49:35.634716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.634700 5226 scope.go:122] "RemoveContainer" containerID="27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c" Oct 30 22:49:35.635922 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.635888 5226 generic.go:356] "Generic (PLEG): container finished" podID="14a37b06-38fb-43df-9284-8209bb6035a6" containerID="d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac" exitCode=0 Oct 30 22:49:35.636029 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.635927 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" Oct 30 22:49:35.636029 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.635944 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" event={"ID":"14a37b06-38fb-43df-9284-8209bb6035a6","Type":"ContainerDied","Data":"d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac"} Oct 30 22:49:35.636029 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.635973 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q" event={"ID":"14a37b06-38fb-43df-9284-8209bb6035a6","Type":"ContainerDied","Data":"1bafa75878fb3f70101c968cba00b5147c0e2a332c8f197f74d514f4591ac538"} Oct 30 22:49:35.644780 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.644762 5226 scope.go:122] "RemoveContainer" containerID="27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c" Oct 30 22:49:35.645051 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:49:35.645034 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c\": container with ID starting with 27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c not found: ID does not exist" containerID="27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c" Oct 30 22:49:35.645114 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.645058 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c"} err="failed to get container status \"27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c\": rpc error: code = NotFound desc = could not find container \"27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c\": container with ID starting with 27c4d355e397ab0a9c86293ca6d810d2d8f6010714cb9a55da3ced23a8e4cf0c not found: ID does not exist" Oct 30 22:49:35.645114 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.645074 5226 scope.go:122] "RemoveContainer" containerID="d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac" Oct 30 22:49:35.653188 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.653171 5226 scope.go:122] "RemoveContainer" containerID="d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac" Oct 30 22:49:35.653460 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:49:35.653438 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac\": container with ID starting with d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac not found: ID does not exist" containerID="d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac" Oct 30 22:49:35.653553 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.653464 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac"} err="failed to get container status \"d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac\": rpc error: code = NotFound desc = could not find container \"d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac\": container with ID starting with d8b645be97dc7dc2c44337767b9eb2bac2785fc95cd527dbd6c178f47361acac not found: ID does not exist" Oct 30 22:49:35.708459 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708430 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-model-cache\") pod \"14a37b06-38fb-43df-9284-8209bb6035a6\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " Oct 30 22:49:35.708647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708470 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-home\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-home\") pod \"14a37b06-38fb-43df-9284-8209bb6035a6\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " Oct 30 22:49:35.708647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708492 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-tmp-dir\") pod \"38d3c91d-e41a-4a37-8367-53540a9f98b2\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " Oct 30 22:49:35.708647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708510 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-tmp-dir\") pod \"14a37b06-38fb-43df-9284-8209bb6035a6\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " Oct 30 22:49:35.708647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708536 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-dshm\") pod \"38d3c91d-e41a-4a37-8367-53540a9f98b2\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " Oct 30 22:49:35.708647 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708589 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d3c91d-e41a-4a37-8367-53540a9f98b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/38d3c91d-e41a-4a37-8367-53540a9f98b2-tls-certs\") pod \"38d3c91d-e41a-4a37-8367-53540a9f98b2\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " Oct 30 22:49:35.709086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708761 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-model-cache" pod "14a37b06-38fb-43df-9284-8209bb6035a6" (UID: "14a37b06-38fb-43df-9284-8209bb6035a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.709086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708793 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-tmp-dir" pod "38d3c91d-e41a-4a37-8367-53540a9f98b2" (UID: "38d3c91d-e41a-4a37-8367-53540a9f98b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.709086 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.708810 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-home" pod "14a37b06-38fb-43df-9284-8209bb6035a6" (UID: "14a37b06-38fb-43df-9284-8209bb6035a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.709255 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709148 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-tmp-dir" pod "14a37b06-38fb-43df-9284-8209bb6035a6" (UID: "14a37b06-38fb-43df-9284-8209bb6035a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.709382 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709351 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d3c91d-e41a-4a37-8367-53540a9f98b2-kube-api-access-9s82v\" (UniqueName: \"kubernetes.io/projected/38d3c91d-e41a-4a37-8367-53540a9f98b2-kube-api-access-9s82v\") pod \"38d3c91d-e41a-4a37-8367-53540a9f98b2\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " Oct 30 22:49:35.709500 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709406 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-model-cache\") pod \"38d3c91d-e41a-4a37-8367-53540a9f98b2\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " Oct 30 22:49:35.709500 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709452 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14a37b06-38fb-43df-9284-8209bb6035a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/14a37b06-38fb-43df-9284-8209bb6035a6-tls-certs\") pod \"14a37b06-38fb-43df-9284-8209bb6035a6\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " Oct 30 22:49:35.709728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709537 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-dshm\") pod \"14a37b06-38fb-43df-9284-8209bb6035a6\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " Oct 30 22:49:35.709728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709579 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14a37b06-38fb-43df-9284-8209bb6035a6-kube-api-access-pdf9t\" (UniqueName: \"kubernetes.io/projected/14a37b06-38fb-43df-9284-8209bb6035a6-kube-api-access-pdf9t\") pod \"14a37b06-38fb-43df-9284-8209bb6035a6\" (UID: \"14a37b06-38fb-43df-9284-8209bb6035a6\") " Oct 30 22:49:35.709728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709638 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-home\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-home\") pod \"38d3c91d-e41a-4a37-8367-53540a9f98b2\" (UID: \"38d3c91d-e41a-4a37-8367-53540a9f98b2\") " Oct 30 22:49:35.709728 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709677 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-model-cache" pod "38d3c91d-e41a-4a37-8367-53540a9f98b2" (UID: "38d3c91d-e41a-4a37-8367-53540a9f98b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.710003 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.709982 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.710065 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.710006 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-home" pod "38d3c91d-e41a-4a37-8367-53540a9f98b2" (UID: "38d3c91d-e41a-4a37-8367-53540a9f98b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.710129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.710011 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.710129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.710093 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.710129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.710109 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.710285 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.710141 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.711514 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.711491 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-dshm" pod "38d3c91d-e41a-4a37-8367-53540a9f98b2" (UID: "38d3c91d-e41a-4a37-8367-53540a9f98b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.711949 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.711926 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d3c91d-e41a-4a37-8367-53540a9f98b2-tls-certs" pod "38d3c91d-e41a-4a37-8367-53540a9f98b2" (UID: "38d3c91d-e41a-4a37-8367-53540a9f98b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:35.712284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.712257 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a37b06-38fb-43df-9284-8209bb6035a6-tls-certs" pod "14a37b06-38fb-43df-9284-8209bb6035a6" (UID: "14a37b06-38fb-43df-9284-8209bb6035a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:35.712284 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.712266 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d3c91d-e41a-4a37-8367-53540a9f98b2-kube-api-access-9s82v" pod "38d3c91d-e41a-4a37-8367-53540a9f98b2" (UID: "38d3c91d-e41a-4a37-8367-53540a9f98b2"). InnerVolumeSpecName "kube-api-access-9s82v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:35.712484 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.712465 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a37b06-38fb-43df-9284-8209bb6035a6-kube-api-access-pdf9t" pod "14a37b06-38fb-43df-9284-8209bb6035a6" (UID: "14a37b06-38fb-43df-9284-8209bb6035a6"). InnerVolumeSpecName "kube-api-access-pdf9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:35.712524 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.712494 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-dshm" pod "14a37b06-38fb-43df-9284-8209bb6035a6" (UID: "14a37b06-38fb-43df-9284-8209bb6035a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:35.748729 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.748693 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:35.810825 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.810797 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.810947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.810827 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38d3c91d-e41a-4a37-8367-53540a9f98b2-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.810947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.810842 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s82v\" (UniqueName: \"kubernetes.io/projected/38d3c91d-e41a-4a37-8367-53540a9f98b2-kube-api-access-9s82v\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.810947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.810857 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14a37b06-38fb-43df-9284-8209bb6035a6-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.810947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.810870 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14a37b06-38fb-43df-9284-8209bb6035a6-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.810947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.810883 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdf9t\" (UniqueName: \"kubernetes.io/projected/14a37b06-38fb-43df-9284-8209bb6035a6-kube-api-access-pdf9t\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:35.810947 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:35.810895 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38d3c91d-e41a-4a37-8367-53540a9f98b2-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:36.049554 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.049523 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c"] Oct 30 22:49:36.053835 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.053808 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/simulator-w2-eb4aa1-kserve-89d49cb87-6rz8c"] Oct 30 22:49:36.121621 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.121585 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r"] Oct 30 22:49:36.128664 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:49:36.128636 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dde2f5_964f_4779_bafe_213551f5e45b.slice/crio-ca548c8d2add18e44844f95ea229f55687e555bdd2eeebddfc384e1d01c6a322 WatchSource:0}: Error finding container ca548c8d2add18e44844f95ea229f55687e555bdd2eeebddfc384e1d01c6a322: Status 404 returned error can't find the container with id ca548c8d2add18e44844f95ea229f55687e555bdd2eeebddfc384e1d01c6a322 Oct 30 22:49:36.155364 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.155336 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q"] Oct 30 22:49:36.157466 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.157443 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-eb4aa1/distinct-2-w2-eb4aa1-kserve-fdcc8794b-bzm7q"] Oct 30 22:49:36.643924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.643831 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" event={"ID":"24dde2f5-964f-4779-bafe-213551f5e45b","Type":"ContainerStarted","Data":"5c9c092dcfa486c9bf65e20dabce8e391cc4ffd9d22fe045411f58aa271da879"} Oct 30 22:49:36.643924 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.643890 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" event={"ID":"24dde2f5-964f-4779-bafe-213551f5e45b","Type":"ContainerStarted","Data":"ca548c8d2add18e44844f95ea229f55687e555bdd2eeebddfc384e1d01c6a322"} Oct 30 22:49:36.753070 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.753007 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" podStartSLOduration=1.752994795 podStartE2EDuration="1.752994795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:36.752466528 +0000 UTC m=+2868.009808237" watchObservedRunningTime="2026-10-30 22:49:36.752994795 +0000 UTC m=+2868.010336529" Oct 30 22:49:36.904654 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.904555 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" path="/var/lib/kubelet/pods/14a37b06-38fb-43df-9284-8209bb6035a6/volumes" Oct 30 22:49:36.905106 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:36.905085 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" path="/var/lib/kubelet/pods/38d3c91d-e41a-4a37-8367-53540a9f98b2/volumes" Oct 30 22:49:37.649557 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:37.649521 5226 generic.go:356] "Generic (PLEG): container finished" podID="24dde2f5-964f-4779-bafe-213551f5e45b" containerID="5c9c092dcfa486c9bf65e20dabce8e391cc4ffd9d22fe045411f58aa271da879" exitCode=0 Oct 30 22:49:37.649946 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:37.649568 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" event={"ID":"24dde2f5-964f-4779-bafe-213551f5e45b","Type":"ContainerDied","Data":"5c9c092dcfa486c9bf65e20dabce8e391cc4ffd9d22fe045411f58aa271da879"} Oct 30 22:49:38.778283 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.778256 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:38.839649 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.839621 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24dde2f5-964f-4779-bafe-213551f5e45b-kube-api-access-f8rrw\" (UniqueName: \"kubernetes.io/projected/24dde2f5-964f-4779-bafe-213551f5e45b-kube-api-access-f8rrw\") pod \"24dde2f5-964f-4779-bafe-213551f5e45b\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " Oct 30 22:49:38.839824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.839719 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-xks-ca\") pod \"24dde2f5-964f-4779-bafe-213551f5e45b\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " Oct 30 22:49:38.839824 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.839741 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-openshift-ca\") pod \"24dde2f5-964f-4779-bafe-213551f5e45b\" (UID: \"24dde2f5-964f-4779-bafe-213551f5e45b\") " Oct 30 22:49:38.840170 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.840146 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-openshift-ca" pod "24dde2f5-964f-4779-bafe-213551f5e45b" (UID: "24dde2f5-964f-4779-bafe-213551f5e45b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:38.841716 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.841691 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24dde2f5-964f-4779-bafe-213551f5e45b-kube-api-access-f8rrw" pod "24dde2f5-964f-4779-bafe-213551f5e45b" (UID: "24dde2f5-964f-4779-bafe-213551f5e45b"). InnerVolumeSpecName "kube-api-access-f8rrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:38.841802 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.841768 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-xks-ca" pod "24dde2f5-964f-4779-bafe-213551f5e45b" (UID: "24dde2f5-964f-4779-bafe-213551f5e45b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:38.941533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.941502 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8rrw\" (UniqueName: \"kubernetes.io/projected/24dde2f5-964f-4779-bafe-213551f5e45b-kube-api-access-f8rrw\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:38.941533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.941529 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-xks-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:38.941533 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:38.941539 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24dde2f5-964f-4779-bafe-213551f5e45b-maas-api-openshift-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:39.660459 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:39.660418 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" event={"ID":"24dde2f5-964f-4779-bafe-213551f5e45b","Type":"ContainerDied","Data":"ca548c8d2add18e44844f95ea229f55687e555bdd2eeebddfc384e1d01c6a322"} Oct 30 22:49:39.660656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:39.660470 5226 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca548c8d2add18e44844f95ea229f55687e555bdd2eeebddfc384e1d01c6a322" Oct 30 22:49:39.660656 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:39.660435 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r" Oct 30 22:49:50.635877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:49:50.635842 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:49:54.413220 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:49:54.413172 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:50:04.755334 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:50:04.755080 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:50:15.255352 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:50:15.255317 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:50:24.410306 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:50:24.410267 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:50:34.350121 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:50:34.350076 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:50:54.409887 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:50:54.409844 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:51:24.409979 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:51:24.409944 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:51:30.451099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:30.451021 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:51:40.715731 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.715694 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd"] Oct 30 22:51:40.716134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716055 5226 cpu_manager.go:414] "containerMap: removing container" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" Oct 30 22:51:40.716134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716065 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" Oct 30 22:51:40.716134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716079 5226 cpu_manager.go:414] "containerMap: removing container" podUID="24dde2f5-964f-4779-bafe-213551f5e45b" containerName="revoke-keys" Oct 30 22:51:40.716134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716084 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24dde2f5-964f-4779-bafe-213551f5e45b" containerName="revoke-keys" Oct 30 22:51:40.716134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716102 5226 cpu_manager.go:414] "containerMap: removing container" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:51:40.716134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716108 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:51:40.716134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716116 5226 cpu_manager.go:414] "containerMap: removing container" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" Oct 30 22:51:40.716134 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716120 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" Oct 30 22:51:40.716426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716148 5226 cpu_manager.go:414] "containerMap: removing container" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:51:40.716426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716153 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57cf677-fcd9-4efc-8a1b-9c3c9770a1e2" containerName="cleanup" Oct 30 22:51:40.716426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716216 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="14a37b06-38fb-43df-9284-8209bb6035a6" containerName="main" Oct 30 22:51:40.716426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716228 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e198900-f639-4ff1-98d3-4f3442f3b785" containerName="cleanup" Oct 30 22:51:40.716426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716237 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cebe87-7505-4ac6-af29-b423530a1ad3" containerName="istio-proxy" Oct 30 22:51:40.716426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716247 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d3c91d-e41a-4a37-8367-53540a9f98b2" containerName="main" Oct 30 22:51:40.716426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716255 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="04469b0c-07cf-497c-9cbc-5233a2e7a08c" containerName="cleanup" Oct 30 22:51:40.716426 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.716260 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="24dde2f5-964f-4779-bafe-213551f5e45b" containerName="revoke-keys" Oct 30 22:51:40.719405 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.719384 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.726779 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.726753 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.726905 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.726830 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.726961 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.726894 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.727019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.726999 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefeea47-4862-4578-8bfa-5101be1a3ba7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.727069 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.727042 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.727171 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.727155 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.727237 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.727194 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.727303 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.727285 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.727346 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.727336 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmf7\" (UniqueName: \"kubernetes.io/projected/cefeea47-4862-4578-8bfa-5101be1a3ba7-kube-api-access-gpmf7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.827877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.827844 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefeea47-4862-4578-8bfa-5101be1a3ba7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.827877 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.827882 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828075 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828174 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828134 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828171 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828257 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828234 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmf7\" (UniqueName: \"kubernetes.io/projected/cefeea47-4862-4578-8bfa-5101be1a3ba7-kube-api-access-gpmf7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828288 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828321 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828371 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828354 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828606 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828578 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828791 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828680 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828886 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828680 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cefeea47-4862-4578-8bfa-5101be1a3ba7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828875 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.828932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.828889 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.831679 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.831657 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.832681 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.832658 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.887693 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.887658 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cefeea47-4862-4578-8bfa-5101be1a3ba7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.889838 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.889810 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmf7\" (UniqueName: \"kubernetes.io/projected/cefeea47-4862-4578-8bfa-5101be1a3ba7-kube-api-access-gpmf7\") pod \"maas-default-gateway-openshift-default-75bf867b6c-lxnwd\" (UID: \"cefeea47-4862-4578-8bfa-5101be1a3ba7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:40.932809 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:40.932775 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd"] Oct 30 22:51:41.032610 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:41.032522 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:41.460644 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:41.460608 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd"] Oct 30 22:51:41.466754 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:51:41.466726 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefeea47_4862_4578_8bfa_5101be1a3ba7.slice/crio-ed1ea6ef68ab62976ef709a53e3bf5b542cbebf48447e5092cd0e31d4ef0e853 WatchSource:0}: Error finding container ed1ea6ef68ab62976ef709a53e3bf5b542cbebf48447e5092cd0e31d4ef0e853: Status 404 returned error can't find the container with id ed1ea6ef68ab62976ef709a53e3bf5b542cbebf48447e5092cd0e31d4ef0e853 Oct 30 22:51:41.468892 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:41.468860 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:51:41.469008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:41.468935 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:51:41.469008 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:41.468981 5226 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:51:42.221366 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:42.221333 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" event={"ID":"cefeea47-4862-4578-8bfa-5101be1a3ba7","Type":"ContainerStarted","Data":"84fead7f3859681ea020a9d756d540b0708b1b6735a4ee010b478a29bd81c1d9"} Oct 30 22:51:42.221800 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:42.221383 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" event={"ID":"cefeea47-4862-4578-8bfa-5101be1a3ba7","Type":"ContainerStarted","Data":"ed1ea6ef68ab62976ef709a53e3bf5b542cbebf48447e5092cd0e31d4ef0e853"} Oct 30 22:51:42.441229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:42.438596 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" podStartSLOduration=2.438572974 podStartE2EDuration="2.438572974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.38081138 +0000 UTC m=+2993.638153089" watchObservedRunningTime="2026-10-30 22:51:42.438572974 +0000 UTC m=+2993.695914685" Oct 30 22:51:43.044305 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:43.044269 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:43.045330 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:43.045298 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lxnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:51:43.045537 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:43.045508 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" podUID="cefeea47-4862-4578-8bfa-5101be1a3ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:51:44.048231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:44.044233 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lxnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:51:44.048231 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:44.044295 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" podUID="cefeea47-4862-4578-8bfa-5101be1a3ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:51:45.043616 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:45.043579 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lxnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:51:45.043903 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:45.043874 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" podUID="cefeea47-4862-4578-8bfa-5101be1a3ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:51:46.043639 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:46.043598 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lxnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:51:46.044254 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:46.044218 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" podUID="cefeea47-4862-4578-8bfa-5101be1a3ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:51:47.037229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:47.033594 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lxnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:51:47.037229 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:47.033675 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" podUID="cefeea47-4862-4578-8bfa-5101be1a3ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:51:48.033583 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:48.033551 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-lxnwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 22:51:48.034043 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:48.033607 5226 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" podUID="cefeea47-4862-4578-8bfa-5101be1a3ba7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 22:51:49.444832 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:49.444789 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:51:49.458019 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:49.457999 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:51:49.640673 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:49.640641 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:49.641096 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:49.641040 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:49.641857 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:49.641836 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-lxnwd" Oct 30 22:51:52.126596 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:51:52.126561 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:51:54.410816 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:51:54.410778 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:52:04.041174 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:52:04.041136 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:52:24.409392 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:52:24.409350 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:52:27.150757 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:52:27.150710 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:52:40.154836 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:52:40.154794 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:52:54.409744 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:52:54.409700 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:53:11.340264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:53:11.340225 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:53:22.486826 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:53:22.486787 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:53:24.411454 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:53:24.411410 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:53:33.980894 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:53:33.980861 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:53:54.409829 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:53:54.409783 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:53:56.943277 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:53:56.943240 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:54:05.421576 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:54:05.421541 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:54:24.409565 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:54:24.409521 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:54:25.040879 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:54:25.040842 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:54:38.252185 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:54:38.252146 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:54:39.049318 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:54:39.049285 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r"] Oct 30 22:54:39.095482 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:54:39.095439 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23580df-2dd8r"] Oct 30 22:54:40.903331 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:54:40.903297 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24dde2f5-964f-4779-bafe-213551f5e45b" path="/var/lib/kubelet/pods/24dde2f5-964f-4779-bafe-213551f5e45b/volumes" Oct 30 22:54:54.410157 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:54:54.410111 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:55:00.674437 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:55:00.674396 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:55:22.767199 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:55:22.767165 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:55:24.419284 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:55:24.419243 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:55:36.053099 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:55:36.053056 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:55:38.855007 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:55:38.854968 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-hfhbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:55:38.855467 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:55:38.855060 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-hfhbd" podUID="82a49049-a92f-4eeb-aa52-ff15da31e664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:55:47.351615 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:55:47.351581 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:55:49.824754 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:55:49.824721 5226 scope.go:122] "RemoveContainer" containerID="5c9c092dcfa486c9bf65e20dabce8e391cc4ffd9d22fe045411f58aa271da879" Oct 30 22:55:54.408933 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:55:54.408893 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:56:09.978156 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:56:09.978080 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:56:20.484377 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:56:20.484347 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:56:24.409186 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:56:24.409152 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:56:37.064503 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:56:37.064467 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:56:49.476087 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:56:49.476057 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:56:49.491129 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:56:49.491107 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 22:56:54.410060 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:56:54.410023 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:56:58.773234 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:56:58.773187 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:57:09.934228 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:57:09.934169 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:57:24.411060 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:57:24.411020 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:57:28.547264 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:57:28.547224 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:57:39.645904 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:57:39.645870 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:57:54.410700 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:57:54.410656 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:58:04.363702 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:04.363666 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:58:18.646784 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.646750 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf"] Oct 30 22:58:18.650038 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.650010 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.671509 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.671479 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859tm\" (UniqueName: \"kubernetes.io/projected/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-kube-api-access-859tm\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.671638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.671532 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-dshm\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.671638 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.671593 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-model-cache\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.671768 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.671701 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-home\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.671768 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.671757 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tls-certs\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.671884 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.671815 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tmp-dir\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.699360 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.699334 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dcc58b\"/\"openshift-service-ca.crt\"" Oct 30 22:58:18.744922 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.744889 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dcc58b\"/\"default-dockercfg-z9qxg\"" Oct 30 22:58:18.744922 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.744888 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-dcc58b\"/\"kube-root-ca.crt\"" Oct 30 22:58:18.745124 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.744896 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-dcc58b\"/\"simulator-single-main-dcc58b-kserve-self-signed-certs\"" Oct 30 22:58:18.746994 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.746969 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf"] Oct 30 22:58:18.772793 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.772763 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-home\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.772916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.772806 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tls-certs\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.772916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.772846 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tmp-dir\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.772916 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.772877 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859tm\" (UniqueName: \"kubernetes.io/projected/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-kube-api-access-859tm\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.773091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.772925 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-dshm\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.773091 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.772948 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-model-cache\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.773263 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.773239 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-home\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.773330 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.773271 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tmp-dir\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.773330 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.773307 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-model-cache\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.775136 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.775114 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-dshm\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.775285 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.775269 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tls-certs\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.830584 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.830547 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-859tm\" (UniqueName: \"kubernetes.io/projected/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-kube-api-access-859tm\") pod \"simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:18.961745 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:18.961707 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:19.345240 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:19.345141 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf"] Oct 30 22:58:19.351985 ip-10-0-141-194 kubenswrapper[5226]: W1030 22:58:19.351958 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6b2b0ae_0c63_49f8_954b_c6a3885883b1.slice/crio-98869c98007eeda250a4d3abf812f5edec6b42a01c78b2f89f56b27d4357584b WatchSource:0}: Error finding container 98869c98007eeda250a4d3abf812f5edec6b42a01c78b2f89f56b27d4357584b: Status 404 returned error can't find the container with id 98869c98007eeda250a4d3abf812f5edec6b42a01c78b2f89f56b27d4357584b Oct 30 22:58:20.266143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:20.266106 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" event={"ID":"e6b2b0ae-0c63-49f8-954b-c6a3885883b1","Type":"ContainerStarted","Data":"737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41"} Oct 30 22:58:20.266143 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:20.266150 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" event={"ID":"e6b2b0ae-0c63-49f8-954b-c6a3885883b1","Type":"ContainerStarted","Data":"98869c98007eeda250a4d3abf812f5edec6b42a01c78b2f89f56b27d4357584b"} Oct 30 22:58:20.375741 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:20.375685 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" podStartSLOduration=2.375671594 podStartE2EDuration="2.375671594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:20.375286756 +0000 UTC m=+3391.632628466" watchObservedRunningTime="2026-10-30 22:58:20.375671594 +0000 UTC m=+3391.633013304" Oct 30 22:58:20.962326 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:20.962290 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:23.447945 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:23.447907 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:58:24.411307 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:58:24.411271 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:58:28.962527 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:28.962492 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:28.975109 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:28.975076 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:29.319253 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:29.319166 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:58:49.362768 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:58:49.362734 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:58:54.410709 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:58:54.410670 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:59:18.478321 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:18.478241 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:59:20.031055 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:20.031020 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf"] Oct 30 22:59:24.410600 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:59:24.410563 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:59:35.050904 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.050840 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" podUID="e6b2b0ae-0c63-49f8-954b-c6a3885883b1" containerName="main" containerID="cri-o://737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41" gracePeriod=45 Oct 30 22:59:35.293224 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.293177 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:59:35.399787 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.399718 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tmp-dir\") pod \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " Oct 30 22:59:35.399787 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.399759 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-kube-api-access-859tm\" (UniqueName: \"kubernetes.io/projected/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-kube-api-access-859tm\") pod \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " Oct 30 22:59:35.399971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.399789 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-dshm\") pod \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " Oct 30 22:59:35.399971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.399805 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-home\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-home\") pod \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " Oct 30 22:59:35.399971 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.399919 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-model-cache\") pod \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " Oct 30 22:59:35.400100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.400024 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tmp-dir" pod "e6b2b0ae-0c63-49f8-954b-c6a3885883b1" (UID: "e6b2b0ae-0c63-49f8-954b-c6a3885883b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:35.400100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.400037 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-home" pod "e6b2b0ae-0c63-49f8-954b-c6a3885883b1" (UID: "e6b2b0ae-0c63-49f8-954b-c6a3885883b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:35.400100 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.400085 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tls-certs\") pod \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\" (UID: \"e6b2b0ae-0c63-49f8-954b-c6a3885883b1\") " Oct 30 22:59:35.400292 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.400155 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-model-cache" pod "e6b2b0ae-0c63-49f8-954b-c6a3885883b1" (UID: "e6b2b0ae-0c63-49f8-954b-c6a3885883b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:35.400429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.400406 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:35.400429 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.400427 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:35.400598 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.400436 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:35.402050 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.402021 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-dshm" pod "e6b2b0ae-0c63-49f8-954b-c6a3885883b1" (UID: "e6b2b0ae-0c63-49f8-954b-c6a3885883b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:59:35.402155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.402028 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-kube-api-access-859tm" pod "e6b2b0ae-0c63-49f8-954b-c6a3885883b1" (UID: "e6b2b0ae-0c63-49f8-954b-c6a3885883b1"). InnerVolumeSpecName "kube-api-access-859tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:59:35.402155 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.402061 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tls-certs" pod "e6b2b0ae-0c63-49f8-954b-c6a3885883b1" (UID: "e6b2b0ae-0c63-49f8-954b-c6a3885883b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:59:35.501317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.501284 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:35.501317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.501312 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-859tm\" (UniqueName: \"kubernetes.io/projected/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-kube-api-access-859tm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:35.501317 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.501322 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b2b0ae-0c63-49f8-954b-c6a3885883b1-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:59:35.629406 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.629360 5226 generic.go:356] "Generic (PLEG): container finished" podID="e6b2b0ae-0c63-49f8-954b-c6a3885883b1" containerID="737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41" exitCode=0 Oct 30 22:59:35.629592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.629411 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" Oct 30 22:59:35.629592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.629445 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" event={"ID":"e6b2b0ae-0c63-49f8-954b-c6a3885883b1","Type":"ContainerDied","Data":"737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41"} Oct 30 22:59:35.629592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.629499 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf" event={"ID":"e6b2b0ae-0c63-49f8-954b-c6a3885883b1","Type":"ContainerDied","Data":"98869c98007eeda250a4d3abf812f5edec6b42a01c78b2f89f56b27d4357584b"} Oct 30 22:59:35.629592 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.629521 5226 scope.go:122] "RemoveContainer" containerID="737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41" Oct 30 22:59:35.638747 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.638728 5226 scope.go:122] "RemoveContainer" containerID="737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41" Oct 30 22:59:35.638996 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:59:35.638975 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41\": container with ID starting with 737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41 not found: ID does not exist" containerID="737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41" Oct 30 22:59:35.639073 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.639008 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41"} err="failed to get container status \"737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41\": rpc error: code = NotFound desc = could not find container \"737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41\": container with ID starting with 737031e8efda5afdf5cf3c8f4940d9f743c5ffc2faac3dc65b2d8e7de1ae8f41 not found: ID does not exist" Oct 30 22:59:35.781660 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.781625 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf"] Oct 30 22:59:35.785932 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:35.785902 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-dcc58b/simulator-single-main-dcc58b-kserve-6cb6686977-vbfkf"] Oct 30 22:59:36.061487 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:36.061407 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 22:59:36.903581 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:36.903544 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b2b0ae-0c63-49f8-954b-c6a3885883b1" path="/var/lib/kubelet/pods/e6b2b0ae-0c63-49f8-954b-c6a3885883b1/volumes" Oct 30 22:59:54.410130 ip-10-0-141-194 kubenswrapper[5226]: E1030 22:59:54.410091 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 22:59:55.673965 ip-10-0-141-194 kubenswrapper[5226]: I1030 22:59:55.673929 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:00:06.864984 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:00:06.864951 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:00:24.410107 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:00:24.410064 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:00:54.409794 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:00:54.409756 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:01:24.410471 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:01:24.410431 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:01:40.078460 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:01:40.078422 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:01:49.508459 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:01:49.508428 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 23:01:49.524739 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:01:49.524717 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 23:01:54.411243 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:01:54.411191 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:01:57.439242 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:01:57.439187 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:02:05.300825 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:02:05.300749 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:02:24.409731 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:02:24.409688 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:02:26.567008 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:02:26.566973 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:02:42.586785 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:02:42.586747 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:02:54.409955 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:02:54.409916 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:03:00.155930 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.155891 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc"] Oct 30 23:03:00.156371 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.156283 5226 cpu_manager.go:414] "containerMap: removing container" podUID="e6b2b0ae-0c63-49f8-954b-c6a3885883b1" containerName="main" Oct 30 23:03:00.156371 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.156296 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b2b0ae-0c63-49f8-954b-c6a3885883b1" containerName="main" Oct 30 23:03:00.156446 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.156392 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b2b0ae-0c63-49f8-954b-c6a3885883b1" containerName="main" Oct 30 23:03:00.159388 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.159366 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.208597 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.208561 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2e092e\"/\"simulator-single-main-2e092e-kserve-self-signed-certs\"" Oct 30 23:03:00.254232 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.254181 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2e092e\"/\"openshift-service-ca.crt\"" Oct 30 23:03:00.254232 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.254216 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2e092e\"/\"default-dockercfg-5fq2x\"" Oct 30 23:03:00.254503 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.254286 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2e092e\"/\"kube-root-ca.crt\"" Oct 30 23:03:00.256033 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.256009 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc"] Oct 30 23:03:00.278724 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.278696 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-model-cache\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.278848 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.278749 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb408fe0-75c5-419b-bce0-121f539906d2-tls-certs\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.278920 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.278902 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-home\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.278964 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.278951 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp6l2\" (UniqueName: \"kubernetes.io/projected/cb408fe0-75c5-419b-bce0-121f539906d2-kube-api-access-mp6l2\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.279055 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.279039 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-tmp-dir\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.279095 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.279078 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-dshm\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.379954 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.379911 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp6l2\" (UniqueName: \"kubernetes.io/projected/cb408fe0-75c5-419b-bce0-121f539906d2-kube-api-access-mp6l2\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.380157 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.379963 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-tmp-dir\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.380157 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.380006 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-dshm\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.380157 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.380028 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-model-cache\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.380157 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.380079 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb408fe0-75c5-419b-bce0-121f539906d2-tls-certs\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.380401 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.380162 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-home\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.380513 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.380485 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-tmp-dir\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.380595 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.380509 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-model-cache\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.380658 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.380602 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-home\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.382481 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.382456 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-dshm\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.382761 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.382737 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb408fe0-75c5-419b-bce0-121f539906d2-tls-certs\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.437509 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.437474 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp6l2\" (UniqueName: \"kubernetes.io/projected/cb408fe0-75c5-419b-bce0-121f539906d2-kube-api-access-mp6l2\") pod \"simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.471657 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.471628 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:00.857487 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:00.857405 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc"] Oct 30 23:03:00.864457 ip-10-0-141-194 kubenswrapper[5226]: W1030 23:03:00.864428 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb408fe0_75c5_419b_bce0_121f539906d2.slice/crio-c55d98d8e7191fcef868524baba273ff7c58bed3fa5d9b1bf0e10f13ceddcb53 WatchSource:0}: Error finding container c55d98d8e7191fcef868524baba273ff7c58bed3fa5d9b1bf0e10f13ceddcb53: Status 404 returned error can't find the container with id c55d98d8e7191fcef868524baba273ff7c58bed3fa5d9b1bf0e10f13ceddcb53 Oct 30 23:03:01.643470 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:01.643410 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" event={"ID":"cb408fe0-75c5-419b-bce0-121f539906d2","Type":"ContainerStarted","Data":"33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de"} Oct 30 23:03:01.643470 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:01.643459 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" event={"ID":"cb408fe0-75c5-419b-bce0-121f539906d2","Type":"ContainerStarted","Data":"c55d98d8e7191fcef868524baba273ff7c58bed3fa5d9b1bf0e10f13ceddcb53"} Oct 30 23:03:01.754525 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:01.754457 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" podStartSLOduration=1.754437978 podStartE2EDuration="1.754437978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:01.753020666 +0000 UTC m=+3673.010362375" watchObservedRunningTime="2026-10-30 23:03:01.754437978 +0000 UTC m=+3673.011779691" Oct 30 23:03:02.472792 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:02.472743 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:02.982112 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:02.982076 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:03:10.472346 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:10.472317 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:10.486155 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:10.486129 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:10.694951 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:10.694921 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:03:24.410046 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:03:24.409997 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:03:33.178032 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:03:33.177951 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:03:54.409815 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:03:54.409768 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:04:01.271128 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:01.271094 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:04:02.635691 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:02.635654 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc"] Oct 30 23:04:17.655665 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:17.655605 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" podUID="cb408fe0-75c5-419b-bce0-121f539906d2" containerName="main" containerID="cri-o://33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de" gracePeriod=45 Oct 30 23:04:17.684224 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:17.684176 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" podUID="cb408fe0-75c5-419b-bce0-121f539906d2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.123:8000/ready\": dial tcp 10.133.0.123:8000: connect: connection refused" Oct 30 23:04:17.902996 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:17.902971 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:04:17.993306 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:17.993273 5226 generic.go:356] "Generic (PLEG): container finished" podID="cb408fe0-75c5-419b-bce0-121f539906d2" containerID="33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de" exitCode=0 Oct 30 23:04:17.993480 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:17.993315 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" Oct 30 23:04:17.993480 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:17.993345 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" event={"ID":"cb408fe0-75c5-419b-bce0-121f539906d2","Type":"ContainerDied","Data":"33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de"} Oct 30 23:04:17.993480 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:17.993374 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc" event={"ID":"cb408fe0-75c5-419b-bce0-121f539906d2","Type":"ContainerDied","Data":"c55d98d8e7191fcef868524baba273ff7c58bed3fa5d9b1bf0e10f13ceddcb53"} Oct 30 23:04:17.993480 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:17.993390 5226 scope.go:122] "RemoveContainer" containerID="33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de" Oct 30 23:04:18.001941 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.001924 5226 scope.go:122] "RemoveContainer" containerID="33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de" Oct 30 23:04:18.002173 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:04:18.002156 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de\": container with ID starting with 33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de not found: ID does not exist" containerID="33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de" Oct 30 23:04:18.002303 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.002183 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de"} err="failed to get container status \"33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de\": rpc error: code = NotFound desc = could not find container \"33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de\": container with ID starting with 33ddc453d5debc403aa18ba24d0269d88a60b498e9c3669537c87e2575dc40de not found: ID does not exist" Oct 30 23:04:18.022856 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.022829 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-tmp-dir\") pod \"cb408fe0-75c5-419b-bce0-121f539906d2\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " Oct 30 23:04:18.022976 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.022886 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb408fe0-75c5-419b-bce0-121f539906d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb408fe0-75c5-419b-bce0-121f539906d2-tls-certs\") pod \"cb408fe0-75c5-419b-bce0-121f539906d2\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " Oct 30 23:04:18.022976 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.022936 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb408fe0-75c5-419b-bce0-121f539906d2-kube-api-access-mp6l2\" (UniqueName: \"kubernetes.io/projected/cb408fe0-75c5-419b-bce0-121f539906d2-kube-api-access-mp6l2\") pod \"cb408fe0-75c5-419b-bce0-121f539906d2\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " Oct 30 23:04:18.023845 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.023644 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-model-cache\") pod \"cb408fe0-75c5-419b-bce0-121f539906d2\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " Oct 30 23:04:18.023845 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.023699 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-dshm\") pod \"cb408fe0-75c5-419b-bce0-121f539906d2\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " Oct 30 23:04:18.023845 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.023787 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-home\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-home\") pod \"cb408fe0-75c5-419b-bce0-121f539906d2\" (UID: \"cb408fe0-75c5-419b-bce0-121f539906d2\") " Oct 30 23:04:18.024173 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.024128 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-model-cache" pod "cb408fe0-75c5-419b-bce0-121f539906d2" (UID: "cb408fe0-75c5-419b-bce0-121f539906d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:18.024412 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.024377 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-home" pod "cb408fe0-75c5-419b-bce0-121f539906d2" (UID: "cb408fe0-75c5-419b-bce0-121f539906d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:18.024536 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.024448 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.024536 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.024520 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-tmp-dir" pod "cb408fe0-75c5-419b-bce0-121f539906d2" (UID: "cb408fe0-75c5-419b-bce0-121f539906d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:18.025623 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.025587 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb408fe0-75c5-419b-bce0-121f539906d2-kube-api-access-mp6l2" pod "cb408fe0-75c5-419b-bce0-121f539906d2" (UID: "cb408fe0-75c5-419b-bce0-121f539906d2"). InnerVolumeSpecName "kube-api-access-mp6l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:04:18.026456 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.026422 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-dshm" pod "cb408fe0-75c5-419b-bce0-121f539906d2" (UID: "cb408fe0-75c5-419b-bce0-121f539906d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:04:18.028593 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.028557 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb408fe0-75c5-419b-bce0-121f539906d2-tls-certs" pod "cb408fe0-75c5-419b-bce0-121f539906d2" (UID: "cb408fe0-75c5-419b-bce0-121f539906d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:04:18.125713 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.125677 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.125713 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.125705 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb408fe0-75c5-419b-bce0-121f539906d2-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.125713 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.125716 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp6l2\" (UniqueName: \"kubernetes.io/projected/cb408fe0-75c5-419b-bce0-121f539906d2-kube-api-access-mp6l2\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.125986 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.125727 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.125986 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.125736 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb408fe0-75c5-419b-bce0-121f539906d2-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:04:18.405407 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.405376 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc"] Oct 30 23:04:18.451965 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.451934 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2e092e/simulator-single-main-2e092e-kserve-d8fc4c597-cb7lc"] Oct 30 23:04:18.904355 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:18.904321 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb408fe0-75c5-419b-bce0-121f539906d2" path="/var/lib/kubelet/pods/cb408fe0-75c5-419b-bce0-121f539906d2/volumes" Oct 30 23:04:21.843559 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:21.843522 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:04:24.410508 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:04:24.410469 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:04:38.550153 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:04:38.550115 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:04:54.409839 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:04:54.409797 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:05:17.548453 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.548368 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb"] Oct 30 23:05:17.548947 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.548758 5226 cpu_manager.go:414] "containerMap: removing container" podUID="cb408fe0-75c5-419b-bce0-121f539906d2" containerName="main" Oct 30 23:05:17.548947 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.548770 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb408fe0-75c5-419b-bce0-121f539906d2" containerName="main" Oct 30 23:05:17.548947 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.548862 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb408fe0-75c5-419b-bce0-121f539906d2" containerName="main" Oct 30 23:05:17.551946 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.551926 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.601008 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.600976 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 23:05:17.646972 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.646929 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 23:05:17.646972 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.646929 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dwlqg\"" Oct 30 23:05:17.646972 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.646937 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4bcacaaa-kserve-self-signed-certs\"" Oct 30 23:05:17.648703 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.648678 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb"] Oct 30 23:05:17.713571 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.713537 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xnrg\" (UniqueName: \"kubernetes.io/projected/a7416065-a026-425f-9ebe-bfc1c9300108-kube-api-access-9xnrg\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.713780 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.713578 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-dshm\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.713780 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.713602 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-tmp-dir\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.713780 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.713747 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7416065-a026-425f-9ebe-bfc1c9300108-tls-certs\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.713913 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.713808 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-model-cache\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.713913 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.713844 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-home\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.815512 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.815423 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xnrg\" (UniqueName: \"kubernetes.io/projected/a7416065-a026-425f-9ebe-bfc1c9300108-kube-api-access-9xnrg\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.815512 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.815470 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-dshm\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.815512 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.815510 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-tmp-dir\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.815805 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.815546 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7416065-a026-425f-9ebe-bfc1c9300108-tls-certs\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.815805 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.815723 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-model-cache\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.815805 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.815781 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-home\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.816029 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.816010 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-tmp-dir\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.816132 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.816110 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-home\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.816423 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.816403 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-model-cache\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.818452 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.818428 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-dshm\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.818646 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.818631 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7416065-a026-425f-9ebe-bfc1c9300108-tls-certs\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:17.873035 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:17.873008 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xnrg\" (UniqueName: \"kubernetes.io/projected/a7416065-a026-425f-9ebe-bfc1c9300108-kube-api-access-9xnrg\") pod \"e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:18.172019 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:18.171987 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:18.547062 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:18.546967 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb"] Oct 30 23:05:18.553807 ip-10-0-141-194 kubenswrapper[5226]: W1030 23:05:18.553778 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7416065_a026_425f_9ebe_bfc1c9300108.slice/crio-ef24b852ce161c2bab6d95bdbdebf8d06185a4c401ba38d1929bfb0a4041fd13 WatchSource:0}: Error finding container ef24b852ce161c2bab6d95bdbdebf8d06185a4c401ba38d1929bfb0a4041fd13: Status 404 returned error can't find the container with id ef24b852ce161c2bab6d95bdbdebf8d06185a4c401ba38d1929bfb0a4041fd13 Oct 30 23:05:19.297501 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:19.297454 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" event={"ID":"a7416065-a026-425f-9ebe-bfc1c9300108","Type":"ContainerStarted","Data":"f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63"} Oct 30 23:05:19.297501 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:19.297510 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" event={"ID":"a7416065-a026-425f-9ebe-bfc1c9300108","Type":"ContainerStarted","Data":"ef24b852ce161c2bab6d95bdbdebf8d06185a4c401ba38d1929bfb0a4041fd13"} Oct 30 23:05:19.408133 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:19.408069 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" podStartSLOduration=2.408050227 podStartE2EDuration="2.408050227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:19.406615552 +0000 UTC m=+3810.663957262" watchObservedRunningTime="2026-10-30 23:05:19.408050227 +0000 UTC m=+3810.665391938" Oct 30 23:05:20.172747 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:20.172707 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:22.191196 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:22.191158 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:05:24.419914 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:05:24.419875 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:05:25.618394 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:25.618352 5226 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-nr4sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 23:05:25.618812 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:25.618414 5226 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-nr4sp" podUID="eeb6098e-aac7-4b16-94fc-e14dd96de102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 23:05:28.172592 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:28.172549 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:28.185306 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:28.185281 5226 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:28.367836 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:28.367806 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:05:40.763003 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:40.762966 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:05:54.411346 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:05:54.411307 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:05:56.156796 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:05:56.156761 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:06:03.076354 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:03.076317 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb"] Oct 30 23:06:07.579296 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:07.579259 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:06:18.096343 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.096282 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" podUID="a7416065-a026-425f-9ebe-bfc1c9300108" containerName="main" containerID="cri-o://f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63" gracePeriod=45 Oct 30 23:06:18.347236 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.347158 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:06:18.387395 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387366 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-model-cache\") pod \"a7416065-a026-425f-9ebe-bfc1c9300108\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " Oct 30 23:06:18.387533 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387464 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7416065-a026-425f-9ebe-bfc1c9300108-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7416065-a026-425f-9ebe-bfc1c9300108-tls-certs\") pod \"a7416065-a026-425f-9ebe-bfc1c9300108\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " Oct 30 23:06:18.387533 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387509 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-home\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-home\") pod \"a7416065-a026-425f-9ebe-bfc1c9300108\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " Oct 30 23:06:18.387614 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387563 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-tmp-dir\") pod \"a7416065-a026-425f-9ebe-bfc1c9300108\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " Oct 30 23:06:18.387614 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387599 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7416065-a026-425f-9ebe-bfc1c9300108-kube-api-access-9xnrg\" (UniqueName: \"kubernetes.io/projected/a7416065-a026-425f-9ebe-bfc1c9300108-kube-api-access-9xnrg\") pod \"a7416065-a026-425f-9ebe-bfc1c9300108\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " Oct 30 23:06:18.387731 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387619 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-model-cache" pod "a7416065-a026-425f-9ebe-bfc1c9300108" (UID: "a7416065-a026-425f-9ebe-bfc1c9300108"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:06:18.387731 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387645 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-dshm\") pod \"a7416065-a026-425f-9ebe-bfc1c9300108\" (UID: \"a7416065-a026-425f-9ebe-bfc1c9300108\") " Oct 30 23:06:18.387850 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387788 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-home" pod "a7416065-a026-425f-9ebe-bfc1c9300108" (UID: "a7416065-a026-425f-9ebe-bfc1c9300108"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:06:18.387908 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387857 5226 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-model-cache\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:18.387908 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.387871 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-tmp-dir" pod "a7416065-a026-425f-9ebe-bfc1c9300108" (UID: "a7416065-a026-425f-9ebe-bfc1c9300108"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:06:18.389792 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.389764 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-dshm" pod "a7416065-a026-425f-9ebe-bfc1c9300108" (UID: "a7416065-a026-425f-9ebe-bfc1c9300108"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:06:18.389914 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.389812 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7416065-a026-425f-9ebe-bfc1c9300108-tls-certs" pod "a7416065-a026-425f-9ebe-bfc1c9300108" (UID: "a7416065-a026-425f-9ebe-bfc1c9300108"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:06:18.389914 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.389870 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7416065-a026-425f-9ebe-bfc1c9300108-kube-api-access-9xnrg" pod "a7416065-a026-425f-9ebe-bfc1c9300108" (UID: "a7416065-a026-425f-9ebe-bfc1c9300108"). InnerVolumeSpecName "kube-api-access-9xnrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:06:18.488810 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.488779 5226 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-home\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:18.488810 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.488808 5226 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-tmp-dir\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:18.489018 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.488823 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xnrg\" (UniqueName: \"kubernetes.io/projected/a7416065-a026-425f-9ebe-bfc1c9300108-kube-api-access-9xnrg\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:18.489018 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.488836 5226 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7416065-a026-425f-9ebe-bfc1c9300108-dshm\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:18.489018 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.488848 5226 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7416065-a026-425f-9ebe-bfc1c9300108-tls-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:06:18.686435 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.686400 5226 generic.go:356] "Generic (PLEG): container finished" podID="a7416065-a026-425f-9ebe-bfc1c9300108" containerID="f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63" exitCode=0 Oct 30 23:06:18.686600 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.686447 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" Oct 30 23:06:18.686600 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.686481 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" event={"ID":"a7416065-a026-425f-9ebe-bfc1c9300108","Type":"ContainerDied","Data":"f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63"} Oct 30 23:06:18.686600 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.686525 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb" event={"ID":"a7416065-a026-425f-9ebe-bfc1c9300108","Type":"ContainerDied","Data":"ef24b852ce161c2bab6d95bdbdebf8d06185a4c401ba38d1929bfb0a4041fd13"} Oct 30 23:06:18.686600 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.686542 5226 scope.go:122] "RemoveContainer" containerID="f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63" Oct 30 23:06:18.695524 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.695504 5226 scope.go:122] "RemoveContainer" containerID="f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63" Oct 30 23:06:18.695773 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:06:18.695755 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63\": container with ID starting with f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63 not found: ID does not exist" containerID="f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63" Oct 30 23:06:18.695815 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.695781 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63"} err="failed to get container status \"f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63\": rpc error: code = NotFound desc = could not find container \"f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63\": container with ID starting with f5d9e53921b58ac951a000af28939a159aee964e1941b1ff4ca220c284d7ce63 not found: ID does not exist" Oct 30 23:06:18.839659 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.839629 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb"] Oct 30 23:06:18.843886 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.843861 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4bcacaaa-kserve-7c8756b6f-swnjb"] Oct 30 23:06:18.903730 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:18.903696 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7416065-a026-425f-9ebe-bfc1c9300108" path="/var/lib/kubelet/pods/a7416065-a026-425f-9ebe-bfc1c9300108/volumes" Oct 30 23:06:21.077852 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:21.077815 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:06:24.410769 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:06:24.410732 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:06:30.656519 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:30.656438 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:06:49.542927 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:49.542898 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 23:06:49.556487 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:49.556469 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 23:06:52.668273 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:52.668233 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:06:53.179373 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.179336 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v"] Oct 30 23:06:53.179950 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.179936 5226 cpu_manager.go:414] "containerMap: removing container" podUID="a7416065-a026-425f-9ebe-bfc1c9300108" containerName="main" Oct 30 23:06:53.179995 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.179953 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7416065-a026-425f-9ebe-bfc1c9300108" containerName="main" Oct 30 23:06:53.180082 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.180071 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7416065-a026-425f-9ebe-bfc1c9300108" containerName="main" Oct 30 23:06:53.184995 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.184969 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.276967 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.276924 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808c667-6a41-4cc9-ba87-6c21267265cc-openshift-service-ca\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.276967 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.276969 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-metrics-certs\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.277175 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.277122 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqcz\" (UniqueName: \"kubernetes.io/projected/5808c667-6a41-4cc9-ba87-6c21267265cc-kube-api-access-mfqcz\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.277254 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.277172 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-maas-api-tls\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.280186 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.280152 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 23:06:53.280186 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.280183 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 23:06:53.280381 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.280185 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 23:06:53.280381 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.280267 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 23:06:53.280381 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.280335 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-v8dzl\"" Oct 30 23:06:53.282104 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.282088 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v"] Oct 30 23:06:53.378284 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.378256 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808c667-6a41-4cc9-ba87-6c21267265cc-openshift-service-ca\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.378480 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.378293 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-metrics-certs\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.378480 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.378369 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqcz\" (UniqueName: \"kubernetes.io/projected/5808c667-6a41-4cc9-ba87-6c21267265cc-kube-api-access-mfqcz\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.378480 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.378409 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-maas-api-tls\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.378979 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.378953 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808c667-6a41-4cc9-ba87-6c21267265cc-openshift-service-ca\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.380906 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.380884 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-maas-api-tls\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.381020 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.381004 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-metrics-certs\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.435779 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.435717 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqcz\" (UniqueName: \"kubernetes.io/projected/5808c667-6a41-4cc9-ba87-6c21267265cc-kube-api-access-mfqcz\") pod \"maas-api-d6bb6f9f5-g9m2v\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.501075 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.501044 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:53.872395 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.872310 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v"] Oct 30 23:06:53.878983 ip-10-0-141-194 kubenswrapper[5226]: W1030 23:06:53.878950 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5808c667_6a41_4cc9_ba87_6c21267265cc.slice/crio-85b9d489486bbe2e39a8a7d56aad4328e9c53e53f66986ce9f2de023f11de5b6 WatchSource:0}: Error finding container 85b9d489486bbe2e39a8a7d56aad4328e9c53e53f66986ce9f2de023f11de5b6: Status 404 returned error can't find the container with id 85b9d489486bbe2e39a8a7d56aad4328e9c53e53f66986ce9f2de023f11de5b6 Oct 30 23:06:53.880524 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.880503 5226 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:06:53.897296 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:53.897269 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" event={"ID":"5808c667-6a41-4cc9-ba87-6c21267265cc","Type":"ContainerStarted","Data":"85b9d489486bbe2e39a8a7d56aad4328e9c53e53f66986ce9f2de023f11de5b6"} Oct 30 23:06:54.410737 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:06:54.410702 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:06:54.903649 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:54.903615 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:06:54.904016 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:54.903659 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" event={"ID":"5808c667-6a41-4cc9-ba87-6c21267265cc","Type":"ContainerStarted","Data":"599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0"} Oct 30 23:06:55.018041 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:06:55.017943 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" podStartSLOduration=1.148998543 podStartE2EDuration="2.017931336s" totalImagesPullingTime="868.932793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:55.017112698 +0000 UTC m=+3906.274454407" watchObservedRunningTime="2026-10-30 23:06:55.017931336 +0000 UTC m=+3906.275273046" Oct 30 23:07:00.914137 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:00.914107 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:07:01.237504 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.237471 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9"] Oct 30 23:07:01.247290 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.247259 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.304121 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.304084 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9"] Oct 30 23:07:01.351917 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.351875 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-maas-api-tls\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.352238 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.351929 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-openshift-service-ca\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.352238 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.352096 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsrcx\" (UniqueName: \"kubernetes.io/projected/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-kube-api-access-wsrcx\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.352238 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.352151 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-metrics-certs\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.453574 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.453531 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-maas-api-tls\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.453789 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.453586 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-openshift-service-ca\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.453789 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.453700 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrcx\" (UniqueName: \"kubernetes.io/projected/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-kube-api-access-wsrcx\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.453789 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.453764 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-metrics-certs\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.454293 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.454268 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-openshift-service-ca\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.456036 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.456016 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-maas-api-tls\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.456152 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.456082 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-metrics-certs\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.511763 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.511691 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrcx\" (UniqueName: \"kubernetes.io/projected/9dfa847d-f2b9-4256-8c16-1c46da1f66a6-kube-api-access-wsrcx\") pod \"maas-api-f7b57794c-bkbl9\" (UID: \"9dfa847d-f2b9-4256-8c16-1c46da1f66a6\") " pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.560088 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.560032 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:01.934611 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:01.934574 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9"] Oct 30 23:07:01.941140 ip-10-0-141-194 kubenswrapper[5226]: W1030 23:07:01.941111 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfa847d_f2b9_4256_8c16_1c46da1f66a6.slice/crio-8f5a8639ab8e1329652e0a967d28e2a058904ea8254c19de382cbc6b5db3c7f3 WatchSource:0}: Error finding container 8f5a8639ab8e1329652e0a967d28e2a058904ea8254c19de382cbc6b5db3c7f3: Status 404 returned error can't find the container with id 8f5a8639ab8e1329652e0a967d28e2a058904ea8254c19de382cbc6b5db3c7f3 Oct 30 23:07:02.941433 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:02.941396 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" event={"ID":"9dfa847d-f2b9-4256-8c16-1c46da1f66a6","Type":"ContainerStarted","Data":"1ddeb9c10af2dfd10491d919383cb0cf903796b92174de61253224b6bfdd5a2f"} Oct 30 23:07:02.941845 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:02.941447 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" event={"ID":"9dfa847d-f2b9-4256-8c16-1c46da1f66a6","Type":"ContainerStarted","Data":"8f5a8639ab8e1329652e0a967d28e2a058904ea8254c19de382cbc6b5db3c7f3"} Oct 30 23:07:02.941845 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:02.941494 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:03.053277 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:03.053156 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" podStartSLOduration=1.245302458 podStartE2EDuration="2.053141922s" totalImagesPullingTime="807.839464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:03.052275598 +0000 UTC m=+3914.309617308" watchObservedRunningTime="2026-10-30 23:07:03.053141922 +0000 UTC m=+3914.310483632" Oct 30 23:07:08.951103 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:08.951074 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f7b57794c-bkbl9" Oct 30 23:07:09.110038 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.110004 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v"] Oct 30 23:07:09.110308 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.110280 5226 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" podUID="5808c667-6a41-4cc9-ba87-6c21267265cc" containerName="maas-api" containerID="cri-o://599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0" gracePeriod=30 Oct 30 23:07:09.366748 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.366720 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:07:09.434051 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.434018 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5808c667-6a41-4cc9-ba87-6c21267265cc-kube-api-access-mfqcz\" (UniqueName: \"kubernetes.io/projected/5808c667-6a41-4cc9-ba87-6c21267265cc-kube-api-access-mfqcz\") pod \"5808c667-6a41-4cc9-ba87-6c21267265cc\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " Oct 30 23:07:09.434225 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.434111 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-metrics-certs\") pod \"5808c667-6a41-4cc9-ba87-6c21267265cc\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " Oct 30 23:07:09.434225 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.434150 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5808c667-6a41-4cc9-ba87-6c21267265cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808c667-6a41-4cc9-ba87-6c21267265cc-openshift-service-ca\") pod \"5808c667-6a41-4cc9-ba87-6c21267265cc\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " Oct 30 23:07:09.434225 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.434190 5226 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-maas-api-tls\") pod \"5808c667-6a41-4cc9-ba87-6c21267265cc\" (UID: \"5808c667-6a41-4cc9-ba87-6c21267265cc\") " Oct 30 23:07:09.434560 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.434526 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5808c667-6a41-4cc9-ba87-6c21267265cc-openshift-service-ca" pod "5808c667-6a41-4cc9-ba87-6c21267265cc" (UID: "5808c667-6a41-4cc9-ba87-6c21267265cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:07:09.436259 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.436233 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-metrics-certs" pod "5808c667-6a41-4cc9-ba87-6c21267265cc" (UID: "5808c667-6a41-4cc9-ba87-6c21267265cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:09.436375 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.436287 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5808c667-6a41-4cc9-ba87-6c21267265cc-kube-api-access-mfqcz" pod "5808c667-6a41-4cc9-ba87-6c21267265cc" (UID: "5808c667-6a41-4cc9-ba87-6c21267265cc"). InnerVolumeSpecName "kube-api-access-mfqcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:07:09.436375 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.436289 5226 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-maas-api-tls" pod "5808c667-6a41-4cc9-ba87-6c21267265cc" (UID: "5808c667-6a41-4cc9-ba87-6c21267265cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:07:09.535270 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.535179 5226 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-metrics-certs\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:09.535270 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.535232 5226 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808c667-6a41-4cc9-ba87-6c21267265cc-openshift-service-ca\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:09.535270 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.535244 5226 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808c667-6a41-4cc9-ba87-6c21267265cc-maas-api-tls\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:09.535270 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.535254 5226 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfqcz\" (UniqueName: \"kubernetes.io/projected/5808c667-6a41-4cc9-ba87-6c21267265cc-kube-api-access-mfqcz\") on node \"ip-10-0-141-194.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 23:07:09.973865 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.973823 5226 generic.go:356] "Generic (PLEG): container finished" podID="5808c667-6a41-4cc9-ba87-6c21267265cc" containerID="599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0" exitCode=0 Oct 30 23:07:09.974337 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.973876 5226 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" Oct 30 23:07:09.974337 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.973886 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" event={"ID":"5808c667-6a41-4cc9-ba87-6c21267265cc","Type":"ContainerDied","Data":"599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0"} Oct 30 23:07:09.974337 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.973936 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v" event={"ID":"5808c667-6a41-4cc9-ba87-6c21267265cc","Type":"ContainerDied","Data":"85b9d489486bbe2e39a8a7d56aad4328e9c53e53f66986ce9f2de023f11de5b6"} Oct 30 23:07:09.974337 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.973966 5226 scope.go:122] "RemoveContainer" containerID="599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0" Oct 30 23:07:09.983100 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.983078 5226 scope.go:122] "RemoveContainer" containerID="599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0" Oct 30 23:07:09.983413 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:07:09.983391 5226 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0\": container with ID starting with 599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0 not found: ID does not exist" containerID="599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0" Oct 30 23:07:09.983494 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:09.983422 5226 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0"} err="failed to get container status \"599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0\": rpc error: code = NotFound desc = could not find container \"599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0\": container with ID starting with 599cb39bc89c55855d0ad37a650df196f74774025bdd823ea053076bc1c60cb0 not found: ID does not exist" Oct 30 23:07:10.127492 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:10.127456 5226 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v"] Oct 30 23:07:10.173568 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:10.173532 5226 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6bb6f9f5-g9m2v"] Oct 30 23:07:10.903876 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:10.903839 5226 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5808c667-6a41-4cc9-ba87-6c21267265cc" path="/var/lib/kubelet/pods/5808c667-6a41-4cc9-ba87-6c21267265cc/volumes" Oct 30 23:07:15.788616 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:07:15.788582 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:07:24.410164 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:07:24.410122 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:07:54.410751 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:07:54.410710 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:08:00.135574 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:08:00.135479 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:08:17.232832 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:08:17.232795 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:08:24.409660 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:08:24.409621 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:08:54.410461 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:08:54.410426 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:09:24.410320 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:09:24.410280 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:09:54.410019 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:09:54.409977 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:10:13.332362 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:10:13.332319 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:10:24.410926 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:10:24.410888 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:10:25.353255 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:10:25.353220 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:10:36.655904 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:10:36.655867 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:10:54.410044 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:10:54.410000 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:11:00.725393 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:00.725310 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:11:24.410503 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:11:24.410466 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:11:34.538888 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:34.538854 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-frwkp"] Oct 30 23:11:45.630848 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:45.630808 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mqnkq_a98769cf-c26c-41fd-8705-f0fac5e64e38/manager/0.log" Oct 30 23:11:46.423730 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:46.423696 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s2ws2_10fa6181-ca6f-4767-8373-7338944442b9/manager/0.log" Oct 30 23:11:47.319369 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:47.319339 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-tsw2f_2b67e0dd-8f53-4ed2-9f23-960b95aa0711/server/0.log" Oct 30 23:11:47.560376 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:47.560332 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-tj2r4_4064f6b8-6293-4953-a7a1-260e0ec8945b/manager/0.log" Oct 30 23:11:48.909472 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:48.909437 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f7b57794c-bkbl9_9dfa847d-f2b9-4256-8c16-1c46da1f66a6/maas-api/0.log" Oct 30 23:11:49.576713 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:49.576687 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 23:11:49.590091 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:49.590070 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlgss_1a72d78e-c521-4dba-b551-5970a7505245/ovn-acl-logging/0.log" Oct 30 23:11:53.086385 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:53.086349 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-frwkp_b9bfbcb2-aa65-460a-bfa1-d8f30929d089/limitador/0.log" Oct 30 23:11:54.411520 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:11:54.411482 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:11:54.871436 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:54.871408 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-784dfbd64d-xnvxq_1df96253-235b-49f1-b70a-a87297020199/kube-auth-proxy/0.log" Oct 30 23:11:56.056430 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:56.056401 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-hfhbd_82a49049-a92f-4eeb-aa52-ff15da31e664/istio-proxy/0.log" Oct 30 23:11:56.377162 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:56.377076 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-lxnwd_cefeea47-4862-4578-8bfa-5101be1a3ba7/istio-proxy/0.log" Oct 30 23:11:56.883872 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:11:56.883827 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-nr4sp_eeb6098e-aac7-4b16-94fc-e14dd96de102/istio-proxy/0.log" Oct 30 23:12:11.949438 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:11.949405 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mqnkq_a98769cf-c26c-41fd-8705-f0fac5e64e38/manager/0.log" Oct 30 23:12:12.531288 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:12.531256 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s2ws2_10fa6181-ca6f-4767-8373-7338944442b9/manager/0.log" Oct 30 23:12:13.283802 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:13.283773 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-tsw2f_2b67e0dd-8f53-4ed2-9f23-960b95aa0711/server/0.log" Oct 30 23:12:13.516055 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:13.516025 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-tj2r4_4064f6b8-6293-4953-a7a1-260e0ec8945b/manager/0.log" Oct 30 23:12:14.735015 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:14.734988 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f7b57794c-bkbl9_9dfa847d-f2b9-4256-8c16-1c46da1f66a6/maas-api/0.log" Oct 30 23:12:18.448290 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:18.448258 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-frwkp_b9bfbcb2-aa65-460a-bfa1-d8f30929d089/limitador/0.log" Oct 30 23:12:20.058957 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:20.058927 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-784dfbd64d-xnvxq_1df96253-235b-49f1-b70a-a87297020199/kube-auth-proxy/0.log" Oct 30 23:12:21.238315 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:21.238283 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-hfhbd_82a49049-a92f-4eeb-aa52-ff15da31e664/istio-proxy/0.log" Oct 30 23:12:21.526371 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:21.526300 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-lxnwd_cefeea47-4862-4578-8bfa-5101be1a3ba7/istio-proxy/0.log" Oct 30 23:12:21.982789 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:12:21.982754 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-nr4sp_eeb6098e-aac7-4b16-94fc-e14dd96de102/istio-proxy/0.log" Oct 30 23:12:24.409272 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:12:24.409234 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:12:54.410279 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:12:54.410240 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:13:24.410281 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:13:24.410246 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:13:45.839786 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:45.839756 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h8pdj_2675c1b0-71d1-40b8-b78c-5ef4a59389d7/global-pull-secret-syncer/0.log" Oct 30 23:13:49.368692 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:49.368658 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4k4pr_20b0ee9f-f99d-472a-a1d8-df44248d9aec/konnectivity-agent/0.log" Oct 30 23:13:51.114841 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:51.114808 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-194.us-west-2.compute.internal_7419d42fafb37e8b2d87d41a6e268c81/haproxy/0.log" Oct 30 23:13:54.411850 ip-10-0-141-194 kubenswrapper[5226]: E1030 23:13:54.411809 5226 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5w4t9" containerName="extractor" Oct 30 23:13:56.525848 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.525811 5226 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj"] Oct 30 23:13:56.526444 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.526424 5226 cpu_manager.go:414] "containerMap: removing container" podUID="5808c667-6a41-4cc9-ba87-6c21267265cc" containerName="maas-api" Oct 30 23:13:56.526529 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.526447 5226 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5808c667-6a41-4cc9-ba87-6c21267265cc" containerName="maas-api" Oct 30 23:13:56.526586 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.526532 5226 memory_manager.go:365] "RemoveStaleState removing state" podUID="5808c667-6a41-4cc9-ba87-6c21267265cc" containerName="maas-api" Oct 30 23:13:56.529784 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.529760 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.578462 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.578427 5226 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q2xpx\"/\"default-dockercfg-xvgbt\"" Oct 30 23:13:56.578462 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.578441 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2xpx\"/\"openshift-service-ca.crt\"" Oct 30 23:13:56.624869 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.624838 5226 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2xpx\"/\"kube-root-ca.crt\"" Oct 30 23:13:56.627229 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.627174 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj"] Oct 30 23:13:56.655363 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.655312 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-proc\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.655565 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.655384 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-sys\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.655565 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.655497 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jws\" (UniqueName: \"kubernetes.io/projected/6ffc8463-4fe5-49d1-8551-93b26e73833c-kube-api-access-l6jws\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.655673 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.655620 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-lib-modules\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.655742 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.655725 5226 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-podres\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757299 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757265 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-podres\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757487 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757311 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-proc\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757487 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757382 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-proc\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757487 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757398 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-podres\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757487 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757428 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-sys\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757487 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757469 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jws\" (UniqueName: \"kubernetes.io/projected/6ffc8463-4fe5-49d1-8551-93b26e73833c-kube-api-access-l6jws\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757738 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757520 5226 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-lib-modules\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757738 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757538 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-sys\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.757738 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.757623 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ffc8463-4fe5-49d1-8551-93b26e73833c-lib-modules\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.819859 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.819782 5226 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jws\" (UniqueName: \"kubernetes.io/projected/6ffc8463-4fe5-49d1-8551-93b26e73833c-kube-api-access-l6jws\") pod \"perf-node-gather-daemonset-rt2rj\" (UID: \"6ffc8463-4fe5-49d1-8551-93b26e73833c\") " pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:56.843353 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:56.843319 5226 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:57.215362 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:57.215326 5226 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj"] Oct 30 23:13:57.221957 ip-10-0-141-194 kubenswrapper[5226]: W1030 23:13:57.221927 5226 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6ffc8463_4fe5_49d1_8551_93b26e73833c.slice/crio-8ebafadee77a7881b3e9d7f62ad83ac2f9e135681aae329a9bd325bbcc4db9cf WatchSource:0}: Error finding container 8ebafadee77a7881b3e9d7f62ad83ac2f9e135681aae329a9bd325bbcc4db9cf: Status 404 returned error can't find the container with id 8ebafadee77a7881b3e9d7f62ad83ac2f9e135681aae329a9bd325bbcc4db9cf Oct 30 23:13:57.929678 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:57.929641 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" event={"ID":"6ffc8463-4fe5-49d1-8551-93b26e73833c","Type":"ContainerStarted","Data":"c3e7ed7c703d31badeb1537502a17c725cf4a2f65abec8827aed12c005a454f8"} Oct 30 23:13:57.930081 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:57.929695 5226 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" event={"ID":"6ffc8463-4fe5-49d1-8551-93b26e73833c","Type":"ContainerStarted","Data":"8ebafadee77a7881b3e9d7f62ad83ac2f9e135681aae329a9bd325bbcc4db9cf"} Oct 30 23:13:57.930081 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:57.929721 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:13:58.041196 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:58.041138 5226 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" podStartSLOduration=2.041123291 podStartE2EDuration="2.041123291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:58.040160601 +0000 UTC m=+4329.297502308" watchObservedRunningTime="2026-10-30 23:13:58.041123291 +0000 UTC m=+4329.298465000" Oct 30 23:13:58.231072 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:13:58.230969 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-frwkp_b9bfbcb2-aa65-460a-bfa1-d8f30929d089/limitador/0.log" Oct 30 23:14:01.381365 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:01.381277 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_204af2f1-c066-4a89-9ddd-120ae692f6c2/alertmanager/0.log" Oct 30 23:14:01.802190 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:01.802160 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_204af2f1-c066-4a89-9ddd-120ae692f6c2/config-reloader/0.log" Oct 30 23:14:02.219291 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:02.219256 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_204af2f1-c066-4a89-9ddd-120ae692f6c2/kube-rbac-proxy-web/0.log" Oct 30 23:14:02.640003 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:02.639931 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_204af2f1-c066-4a89-9ddd-120ae692f6c2/kube-rbac-proxy/0.log" Oct 30 23:14:03.051751 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:03.051717 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_204af2f1-c066-4a89-9ddd-120ae692f6c2/kube-rbac-proxy-metric/0.log" Oct 30 23:14:03.470936 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:03.470906 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_204af2f1-c066-4a89-9ddd-120ae692f6c2/prom-label-proxy/0.log" Oct 30 23:14:03.882910 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:03.882806 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_204af2f1-c066-4a89-9ddd-120ae692f6c2/init-config-reloader/0.log" Oct 30 23:14:03.944874 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:03.944847 5226 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q2xpx/perf-node-gather-daemonset-rt2rj" Oct 30 23:14:06.098395 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:06.098365 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-cbb75f5c9-k9gj6_3058b5f8-ae0e-4193-b4dd-6d4abace3282/metrics-server/0.log" Oct 30 23:14:08.237964 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:08.237936 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7dlr_3f522003-5ca0-45d8-b32e-b52db6916aca/node-exporter/0.log" Oct 30 23:14:08.685346 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:08.685321 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7dlr_3f522003-5ca0-45d8-b32e-b52db6916aca/kube-rbac-proxy/0.log" Oct 30 23:14:09.095955 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:09.095878 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7dlr_3f522003-5ca0-45d8-b32e-b52db6916aca/init-textfile/0.log" Oct 30 23:14:12.079075 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:12.079049 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81841a12-765d-4fb9-b7e3-1c6414aaa29f/prometheus/0.log" Oct 30 23:14:12.491866 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:12.491840 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81841a12-765d-4fb9-b7e3-1c6414aaa29f/config-reloader/0.log" Oct 30 23:14:12.904126 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:12.904048 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81841a12-765d-4fb9-b7e3-1c6414aaa29f/thanos-sidecar/0.log" Oct 30 23:14:13.315110 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:13.315078 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81841a12-765d-4fb9-b7e3-1c6414aaa29f/kube-rbac-proxy-web/0.log" Oct 30 23:14:13.729588 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:13.729557 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81841a12-765d-4fb9-b7e3-1c6414aaa29f/kube-rbac-proxy/0.log" Oct 30 23:14:14.141245 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:14.141153 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81841a12-765d-4fb9-b7e3-1c6414aaa29f/kube-rbac-proxy-thanos/0.log" Oct 30 23:14:14.562640 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:14.562604 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81841a12-765d-4fb9-b7e3-1c6414aaa29f/init-config-reloader/0.log" Oct 30 23:14:16.251038 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:16.251008 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc79ffd54-xnz4d_9dce0456-dccb-4e0f-bdbd-978371813dc4/telemeter-client/0.log" Oct 30 23:14:16.662316 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:16.662236 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc79ffd54-xnz4d_9dce0456-dccb-4e0f-bdbd-978371813dc4/reload/0.log" Oct 30 23:14:17.083430 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:17.083398 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc79ffd54-xnz4d_9dce0456-dccb-4e0f-bdbd-978371813dc4/kube-rbac-proxy/0.log" Oct 30 23:14:17.528286 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:17.528255 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665bfc67b6-srdl5_10f4cba6-57dc-4397-a602-21a3bad21b82/thanos-query/0.log" Oct 30 23:14:17.969616 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:17.969585 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665bfc67b6-srdl5_10f4cba6-57dc-4397-a602-21a3bad21b82/kube-rbac-proxy-web/0.log" Oct 30 23:14:18.381238 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:18.381148 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665bfc67b6-srdl5_10f4cba6-57dc-4397-a602-21a3bad21b82/kube-rbac-proxy/0.log" Oct 30 23:14:18.792273 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:18.792245 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665bfc67b6-srdl5_10f4cba6-57dc-4397-a602-21a3bad21b82/prom-label-proxy/0.log" Oct 30 23:14:19.210298 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:19.210265 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665bfc67b6-srdl5_10f4cba6-57dc-4397-a602-21a3bad21b82/kube-rbac-proxy-rules/0.log" Oct 30 23:14:19.624784 ip-10-0-141-194 kubenswrapper[5226]: I1030 23:14:19.624710 5226 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665bfc67b6-srdl5_10f4cba6-57dc-4397-a602-21a3bad21b82/kube-rbac-proxy-metrics/0.log"